[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[
https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530869#comment-16530869
]
Hadoop QA commented on HBASE-20691:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
26s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
34s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}115m
16s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20691 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930054/HBASE-20691.v7.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3a1ac491687a 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 13e4578be8 |
| maven |
[jira] [Commented] (HBASE-20829) Remove the addFront assertion in MasterProcedureScheduler.doAdd
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530819#comment-16530819
]
Hadoop QA commented on HBASE-20829:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
57s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} hbase-server: The patch generated 0 new + 6
unchanged - 1 fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 40s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}179m 42s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}237m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHStore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20829 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930036/HBASE-20829-v1.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4c6a9d94e8f5 4.4.0-98-generic #121-Ubuntu SMP Tue Oct 10
14:24:03 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / cfdabe9267 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/13474/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Updated] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-20193: - Attachment: HBASE-20193.master.013.patch > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch, HBASE-20193.master.004.patch, > HBASE-20193.master.005.patch, HBASE-20193.master.006.patch, > HBASE-20193.master.006.patch, HBASE-20193.master.007.patch, > HBASE-20193.master.008.patch, HBASE-20193.master.009.patch, > HBASE-20193.master.010.patch, HBASE-20193.master.011.patch, > HBASE-20193.master.012.patch, HBASE-20193.master.013.patch, > replication_rs_2.jpg > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20839) Fallback to FSHLog if we can not instantiated AsyncFSWAL when user does not specify AsyncFSWAL explicitly
Duo Zhang created HBASE-20839: - Summary: Fallback to FSHLog if we can not instantiated AsyncFSWAL when user does not specify AsyncFSWAL explicitly Key: HBASE-20839 URL: https://issues.apache.org/jira/browse/HBASE-20839 Project: HBase Issue Type: Sub-task Components: wal Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530781#comment-16530781
]
Duo Zhang commented on HBASE-20244:
---
We should probably fall back to FSHLog if AsyncFSWAL can not be instantiated if
there is wal.provider config provided, or the configured value is default. Let
me open a new issue to land it, maybe a bit ugly...
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20838) Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils
Yu Li created HBASE-20838: - Summary: Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils Key: HBASE-20838 URL: https://issues.apache.org/jira/browse/HBASE-20838 Project: HBase Issue Type: Test Reporter: Yu Li Assignee: Yu Li As per [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662] in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the test should be in TestCommonFSUtils -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20791) RSGroupBasedLoadBalancer#setClusterMetrics should pass ClusterMetrics to its internalBalancer
[ https://issues.apache.org/jira/browse/HBASE-20791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530767#comment-16530767 ] Reid Chan commented on HBASE-20791: --- ping [~javaman_chen], would you please also provide a patch for branch-1? > RSGroupBasedLoadBalancer#setClusterMetrics should pass ClusterMetrics to its > internalBalancer > - > > Key: HBASE-20791 > URL: https://issues.apache.org/jira/browse/HBASE-20791 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0, 2.0.0 >Reporter: chenxu >Assignee: chenxu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 20791-master-v2.patch, HBASE-20791-branch-2-v1.patch, > HBASE-20791-master-v1.patch, HBASE-20791-master-v3.patch, > HBASE-20791-master-v4.patch > > > RSGroupBasedLoadBalancer#setClusterMetrics should pass ClusterMetrics to it’s > internalBalancer, Or the StochasticLoadBalancer(internal balancer) will lose > it's Up-to-date RegionLoads info, and effect the balance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-6028: Attachment: HBASE-6028.master.008.patch > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > Attachments: HBASE-6028.master.007.patch, > HBASE-6028.master.008.patch, HBASE-6028.master.008.patch > > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530763#comment-16530763 ] Umesh Agashe commented on HBASE-6028: - 2 of the rubocop messages are about 'Line too long'. Rest of the ruby code has 100 chars wide lines, rubocop expects 80. These messages can be ignored. Unit test failure 'TestSyncReplicationStandbyKillMaster' doesn't seem to be related to the changes. Retrying the build. > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > Attachments: HBASE-6028.master.007.patch, HBASE-6028.master.008.patch > > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530760#comment-16530760 ] Duo Zhang commented on HBASE-20193: --- No big problem. Will fix the problem when committing. > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch, HBASE-20193.master.004.patch, > HBASE-20193.master.005.patch, HBASE-20193.master.006.patch, > HBASE-20193.master.006.patch, HBASE-20193.master.007.patch, > HBASE-20193.master.008.patch, HBASE-20193.master.009.patch, > HBASE-20193.master.010.patch, HBASE-20193.master.011.patch, > HBASE-20193.master.012.patch, replication_rs_2.jpg > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Make IDE configuration for import order match that in our checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20837:
Summary: Make IDE configuration for import order match that in our
checkstyle module (was: Sync import orders between IDE and checkstyle module)
> Make IDE configuration for import order match that in our checkstyle module
> ---
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
> Components: community
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530758#comment-16530758 ] Yu Li commented on HBASE-20691: --- Uploaded the v7 patch to review board for better and more thorough review. Hopefully we could close this one in another one or two rounds of review, so we could make it into release 2.1. Thanks. > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Blocker > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch, HBASE-20691.v7.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20837:
Component/s: community
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
> Components: community
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20837:
Fix Version/s: 2.2.0
1.5.0
3.0.0
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
> Components: community
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530756#comment-16530756
]
Sean Busbey commented on HBASE-20244:
-
Please include a note in the Hadoop section of the ref guide calling out that
users of Hadoop versions with the causes of these breakages will need to change
their WAL provider to {{filesystem}}.
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530756#comment-16530756
]
Sean Busbey edited comment on HBASE-20244 at 7/3/18 3:02 AM:
-
Please include a note in the Hadoop section of the ref guide calling out that
users of Hadoop versions with the causes of these breakages will need to change
their WAL provider to {{filesystem}} (for HBase 2.y.z versions prior to this
fix).
was (Author: busbey):
Please include a note in the Hadoop section of the ref guide calling out that
users of Hadoop versions with the causes of these breakages will need to change
their WAL provider to {{filesystem}}.
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530751#comment-16530751 ] Duo Zhang commented on HBASE-20691: --- This is an important one as the storage policy config does not work. Marked as blocker. > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Blocker > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch, HBASE-20691.v7.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20691: -- Priority: Blocker (was: Minor) > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Blocker > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch, HBASE-20691.v7.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20691: -- Fix Version/s: (was: 2.2.0) 2.1.0 > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Minor > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch, HBASE-20691.v7.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-20691: -- Attachment: HBASE-20691.v7.patch > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Minor > Fix For: 1.5.0, 2.2.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch, HBASE-20691.v7.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20826) Truncate responseInfo attributes on RpcServer WARN messages
[
https://issues.apache.org/jira/browse/HBASE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20826:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2.0+.
> Truncate responseInfo attributes on RpcServer WARN messages
> ---
>
> Key: HBASE-20826
> URL: https://issues.apache.org/jira/browse/HBASE-20826
> Project: HBase
> Issue Type: Improvement
> Components: rpc
>Reporter: Sergey Soldatov
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20826.001.branch-2.0.patch,
> HBASE-20826.002.branch-2.0.patch
>
>
> With Phoenix in the picture, dumping the {{Call}} protobuf to the RS log can
> get *really* chatty, real fast. Notably, some serialized filters just spam
> the log with binary garbage.
> Let's add an upper-limit to the length of params we'll put out at WARN, and
> leave the full content for TRACE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530743#comment-16530743 ] Yu Li commented on HBASE-20691: --- bq. Aren't we expecting an IOException because the storage policy setting is a legit one and so it goes to the test FS implementation? The test FS implementation could assure that right after it entering the setStorage method the exception would be thrown and there won't be any path validation > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Minor > Fix For: 1.5.0, 2.2.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20448) update ref guide to expressly use shaded clients for examples
[ https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20448: -- Fix Version/s: (was: 2.2.0) 2.1.1 > update ref guide to expressly use shaded clients for examples > - > > Key: HBASE-20448 > URL: https://issues.apache.org/jira/browse/HBASE-20448 > Project: HBase > Issue Type: Sub-task > Components: Client, documentation, mapreduce >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 2.1.1 > > > the whole mapreduce section, especially, should be using the shaded version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20331) clean up shaded packaging for 2.1
[ https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20331: -- Fix Version/s: 2.1.1 > clean up shaded packaging for 2.1 > - > > Key: HBASE-20331 > URL: https://issues.apache.org/jira/browse/HBASE-20331 > Project: HBase > Issue Type: Umbrella > Components: Client, mapreduce, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1 > > > polishing pass on shaded modules for 2.0 based on trying to use them in more > contexts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20244:
Affects Version/s: 2.0.0
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20244:
Affects Version/s: 2.0.1
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20244:
Component/s: wal
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Affects Versions: 2.0.0, 2.0.1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20812) Add defaults to Table Interface so implementors don't have to
[ https://issues.apache.org/jira/browse/HBASE-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20812: -- Fix Version/s: 2.2.0 > Add defaults to Table Interface so implementors don't have to > - > > Key: HBASE-20812 > URL: https://issues.apache.org/jira/browse/HBASE-20812 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0 > > Attachments: > 0001-HBASE-20812-Add-defaults-to-Table-Interface-so-imple.patch, 20812.txt, > HBASE-20812.branch-2.0.001.patch, HBASE-20812.branch-2.0.002.patch, > HBASE-20812.branch-2.0.003.patch > > > Lets add default implementaitons -- even if they are just throw > NotImplementedException -- to our Table Interface now we are up on jdk8. > Table implementations are how the likes of hbase-indexer modify hbase --via > a publically supported API -- and I notice that the kafka proxy now goes the > same route. Typically, these customizations are only interested in one or two > methods of Table adding in their own implementations but they have to supply > implementations for all Table methods in their override. Lets help them out > by adding defaults (I had a patch but lost it...). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20812) Add defaults to Table Interface so implementors don't have to
[
https://issues.apache.org/jira/browse/HBASE-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530706#comment-16530706
]
Chia-Ping Tsai commented on HBASE-20812:
{quote}Let's do it in another issue [~chia7712] so it will not block the 2.1.0
release?
{quote}
+1
> Add defaults to Table Interface so implementors don't have to
> -
>
> Key: HBASE-20812
> URL: https://issues.apache.org/jira/browse/HBASE-20812
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments:
> 0001-HBASE-20812-Add-defaults-to-Table-Interface-so-imple.patch, 20812.txt,
> HBASE-20812.branch-2.0.001.patch, HBASE-20812.branch-2.0.002.patch,
> HBASE-20812.branch-2.0.003.patch
>
>
> Lets add default implementaitons -- even if they are just throw
> NotImplementedException -- to our Table Interface now we are up on jdk8.
> Table implementations are how the likes of hbase-indexer modify hbase --via
> a publically supported API -- and I notice that the kafka proxy now goes the
> same route. Typically, these customizations are only interested in one or two
> methods of Table adding in their own implementations but they have to supply
> implementations for all Table methods in their override. Lets help them out
> by adding defaults (I had a patch but lost it...).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20244:
--
Fix Version/s: 2.2.0
2.1.0
3.0.0
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20244:
--
Priority: Blocker (was: Major)
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Blocker
> Fix For: 2.0.2
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20812) Add defaults to Table Interface so implementors don't have to
[ https://issues.apache.org/jira/browse/HBASE-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20812: -- Resolution: Fixed Status: Resolved (was: Patch Available) Let's do it in another issue [~chia7712] so it will not block the 2.1.0 release? As this is not a blocker and it is holiday in America(Independence Day perhaps?) > Add defaults to Table Interface so implementors don't have to > - > > Key: HBASE-20812 > URL: https://issues.apache.org/jira/browse/HBASE-20812 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: > 0001-HBASE-20812-Add-defaults-to-Table-Interface-so-imple.patch, 20812.txt, > HBASE-20812.branch-2.0.001.patch, HBASE-20812.branch-2.0.002.patch, > HBASE-20812.branch-2.0.003.patch > > > Lets add default implementaitons -- even if they are just throw > NotImplementedException -- to our Table Interface now we are up on jdk8. > Table implementations are how the likes of hbase-indexer modify hbase --via > a publically supported API -- and I notice that the kafka proxy now goes the > same route. Typically, these customizations are only interested in one or two > methods of Table adding in their own implementations but they have to supply > implementations for all Table methods in their override. Lets help them out > by adding defaults (I had a patch but lost it...). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20691: -- Fix Version/s: (was: 2.1.0) 2.2.0 > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Minor > Fix For: 1.5.0, 2.2.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch, > HBASE-20691.v6.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20188) [TESTING] Performance
[ https://issues.apache.org/jira/browse/HBASE-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20188: -- Fix Version/s: (was: 2.1.0) 2.2.0 > [TESTING] Performance > - > > Key: HBASE-20188 > URL: https://issues.apache.org/jira/browse/HBASE-20188 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Blocker > Fix For: 3.0.0, 2.2.0 > > Attachments: CAM-CONFIG-V01.patch, HBASE-20188-xac.sh, > HBASE-20188.sh, HBase 2.0 performance evaluation - 8GB(1).pdf, HBase 2.0 > performance evaluation - 8GB.pdf, HBase 2.0 performance evaluation - Basic vs > None_ system settings.pdf, HBase 2.0 performance evaluation - throughput > SSD_HDD.pdf, ITBLL2.5B_1.2.7vs2.0.0_cpu.png, > ITBLL2.5B_1.2.7vs2.0.0_gctime.png, ITBLL2.5B_1.2.7vs2.0.0_iops.png, > ITBLL2.5B_1.2.7vs2.0.0_load.png, ITBLL2.5B_1.2.7vs2.0.0_memheap.png, > ITBLL2.5B_1.2.7vs2.0.0_memstore.png, ITBLL2.5B_1.2.7vs2.0.0_ops.png, > ITBLL2.5B_1.2.7vs2.0.0_ops_NOT_summing_regions.png, YCSB_CPU.png, > YCSB_GC_TIME.png, YCSB_IN_MEMORY_COMPACTION=NONE.ops.png, YCSB_MEMSTORE.png, > YCSB_OPs.png, YCSB_in-memory-compaction=NONE.ops.png, YCSB_load.png, > flamegraph-1072.1.svg, flamegraph-1072.2.svg, hbase-env.sh, hbase-site.xml, > hbase-site.xml, hits.png, hits_with_fp_scheduler.png, > lock.127.workloadc.20180402T200918Z.svg, > lock.2.memsize2.c.20180403T160257Z.svg, perregion.png, run_ycsb.sh, > total.png, tree.txt, workloadx, workloadx > > > How does 2.0.0 compare to old versions? Is it faster, slower? There is rumor > that it is much slower, that the problem is the asyncwal writing. Does > in-memory compaction slow us down or speed us up? What happens when you > enable offheaping? > Keep notes here in this umbrella issue. Need to be able to say something > about perf when 2.0.0 ships. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20835) Document how to get replication reporting
[ https://issues.apache.org/jira/browse/HBASE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530675#comment-16530675 ] Duo Zhang commented on HBASE-20835: --- that means ithere is no peer if the '/hbase/replication/rs' is not there, then your reporting tool could just do a sleep and try again later. > Document how to get replication reporting > - > > Key: HBASE-20835 > URL: https://issues.apache.org/jira/browse/HBASE-20835 > Project: HBase > Issue Type: Task > Components: Replication >Affects Versions: 2.1.0 >Reporter: Mike Drob >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.1.0 > > > Based on my questions at the tail end of HBASE-19543 > bq. We have some tooling that checks on replication queues and reads the > znode as the source of truth. When replication is disabled, it's expected > that the node was still there, but just empty. Is there a better way to get > this same information? > I understand that with table based replication it doesn't make sense to check > ZK for status. However, losing the ability to inspect the data and get > information is a tough hit for operators. Do we have APIs that expose the > same sort of metrics? > bq. how many peers/queues, queue size, position in the queue, and age of last > op > Assigning to you for now, Duo, since you were both primary implementor and RM > for 2.1.0 and I'm not sure who else would know the answers. If the docs > already exist, then nothing to do but we should include them in the RN. Maybe > this will need additional code, but I hope it's already there and is > something we can write a workaround for. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20835) Document how to get replication reporting
[ https://issues.apache.org/jira/browse/HBASE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530669#comment-16530669 ] Duo Zhang commented on HBASE-20835: --- I think the only problem is that we do not create the zk node immediately after creating the peer? There is no other difference. To be honest I do not fully understand what is the real problem for you, as there is no way to disable replication in 2.0? > Document how to get replication reporting > - > > Key: HBASE-20835 > URL: https://issues.apache.org/jira/browse/HBASE-20835 > Project: HBase > Issue Type: Task > Components: Replication >Affects Versions: 2.1.0 >Reporter: Mike Drob >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.1.0 > > > Based on my questions at the tail end of HBASE-19543 > bq. We have some tooling that checks on replication queues and reads the > znode as the source of truth. When replication is disabled, it's expected > that the node was still there, but just empty. Is there a better way to get > this same information? > I understand that with table based replication it doesn't make sense to check > ZK for status. However, losing the ability to inspect the data and get > information is a tough hit for operators. Do we have APIs that expose the > same sort of metrics? > bq. how many peers/queues, queue size, position in the queue, and age of last > op > Assigning to you for now, Duo, since you were both primary implementor and RM > for 2.1.0 and I'm not sure who else would know the answers. If the docs > already exist, then nothing to do but we should include them in the RN. Maybe > this will need additional code, but I hope it's already there and is > something we can write a workaround for. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20829) Remove the addFront assertion in MasterProcedureScheduler.doAdd
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20829:
--
Fix Version/s: 2.2.0
2.1.0
> Remove the addFront assertion in MasterProcedureScheduler.doAdd
> ---
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829-v1.patch,
> HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20244) NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient
[
https://issues.apache.org/jira/browse/HBASE-20244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530651#comment-16530651
]
Duo Zhang commented on HBASE-20244:
---
OK this one has also been ported to 2.8.4.
Let me take a look.
> NoSuchMethodException when retrieving private method
> decryptEncryptedDataEncryptionKey from DFSClient
> -
>
> Key: HBASE-20244
> URL: https://issues.apache.org/jira/browse/HBASE-20244
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.2
>
> Attachments: 20244.v1.txt, 20244.v1.txt, 20244.v1.txt
>
>
> I was running unit test against hadoop 3.0.1 RC and saw the following in test
> output:
> {code}
> ERROR [RS-EventLoopGroup-3-3]
> asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(267): Couldn't properly
> initialize access to HDFS internals. Please update your WAL Provider to not
> make use of the 'asyncfs' provider. See HBASE-16110 for more information.
> java.lang.NoSuchMethodException:
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
> at java.lang.Class.getDeclaredMethod(Class.java:2130)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
> at
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
> at
> org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
> {code}
> The private method was moved by HDFS-12574 to HdfsKMSUtil with different
> signature.
> To accommodate the above method movement, it seems we need to call the
> following method of DFSClient :
> {code}
> public KeyProvider getKeyProvider() throws IOException {
> {code}
> Since the new decryptEncryptedDataEncryptionKey method has this signature:
> {code}
> static KeyVersion decryptEncryptedDataEncryptionKey(FileEncryptionInfo
> feInfo, KeyProvider keyProvider) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20542: -- Component/s: in-memory-compaction > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task > Components: in-memory-compaction >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-20542: --- This breaks TestWriteHeavyIncrementObserverWithMemStoreCompaction. The UT assumes that with in-memory compaction, we can do all the aggregation in memory so there will be no store file generated, but with the patch here we do have some store files generated. So what's the problem? With this patch we slow down the in memory compaction? > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20542) Better heap utilization for IMC with MSLABs
[ https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20542: -- Fix Version/s: 2.2.0 3.0.0 > Better heap utilization for IMC with MSLABs > --- > > Key: HBASE-20542 > URL: https://issues.apache.org/jira/browse/HBASE-20542 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20542.branch-2.001.patch, > HBASE-20542.branch-2.003.patch, HBASE-20542.branch-2.004.patch, > HBASE-20542.branch-2.005.patch, HBASE-20542.master.003.patch, run.sh, > workloada, workloadc, workloadx, workloady > > > Following HBASE-20188 we realized in-memory compaction combined with MSLABs > may suffer from heap under-utilization due to internal fragmentation. This > jira presents a solution to circumvent this problem. The main idea is to have > each update operation check if it will cause overflow in the active segment > *before* it is writing the new value (instead of checking the size after the > write is completed), and if it is then the active segment is atomically > swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to > the compaction pipeline. Later on the IMC deamon will run its compaction > operation (flatten index/merge indices/data compaction) in the background. > Some subtle concurrency issues should be handled with care. We next elaborate > on them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20829) Remove the addFront assertion in MasterProcedureScheduler.doAdd
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530635#comment-16530635
]
Duo Zhang commented on HBASE-20829:
---
Keep the yield method as is. I think there is a problem that for a table
procedure, we will not hold the lock all the time, so if we add the procedure
to the back and then another procedure starts, things may be messed up...
This is another problem we need to fix...
> Remove the addFront assertion in MasterProcedureScheduler.doAdd
> ---
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829-v1.patch,
> HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20829) Remove the addFront assertion in MasterProcedureScheduler.doAdd
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20829:
--
Attachment: HBASE-20829-v1.patch
> Remove the addFront assertion in MasterProcedureScheduler.doAdd
> ---
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829-v1.patch,
> HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20829) Remove the addFront assertion in MasterProcedureScheduler.doAdd
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20829:
--
Summary: Remove the addFront assertion in MasterProcedureScheduler.doAdd
(was: TestSyncReplicationStandbyKillRS is flakey)
> Remove the addFront assertion in MasterProcedureScheduler.doAdd
> ---
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[
https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530620#comment-16530620
]
Hadoop QA commented on HBASE-6028:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
39s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
3s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} root: The patch generated 0 new + 636 unchanged - 5
fixed = 636 total (was 641) {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
10s{color} | {color:red} The patch generated 3 new + 411 unchanged - 0 fixed =
414 total (was 411) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 3s{color} | {color:orange} The patch generated 8 new + 725 unchanged - 0
fixed = 733 total (was 725) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
26s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
59s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 6s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
5m 32s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} |
[jira] [Comment Edited] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530581#comment-16530581
]
Tak Lon (Stephen) Wu edited comment on HBASE-20837 at 7/2/18 11:59 PM:
---
after I dug into the problem, I figured that it's because the community should
have two main sets of developers using Eclipse and IntelliJ, which each tool
has a different way importing formatter and `Organize Imports`/ Import Layout
(Eclipse has to export the `Organize Imports` separately instead of having it
within hbase_eclipse_formatter.xml)
I will send a message to dev list, if we confirm the import layouts as
proposed, we could add the following(s) to the dev-support (at least for
Eclipse) to align the developer editor to the checkstyle module.
1. add *.importorder with dev-support for Eclipse
2. (optional) add *.xml for IntelliJ support for formatter and Import Layout
Will see how it goes but at least we have a Jira down for discussion and as a
record for this issue.
was (Author: taklwu):
after I dug into the problem, I figured that it's because the community should
have two main sets of developers using Eclipse and IntelliJ, which each tool
has a different way importing formatter and `Organize Imports`/ Import Layout
(Eclipse has to export the `Organize Imports` separately instead of having it
within hbase_eclipse_formatter.xml)
I will send a message to community, if we confirm the import layouts as
proposed, we could add the following the dev-support (at least for the Eclipse)
to align the developer editor.
1. add *.importorder with dev-support for Eclipse
2. (optional) add *.xml for IntelliJ support for formatter and Import Layout
Will see how it goes but at least we have a Jira down for discussion and as a
record for this issue.
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530581#comment-16530581
]
Tak Lon (Stephen) Wu edited comment on HBASE-20837 at 7/2/18 11:58 PM:
---
after I dug into the problem, I figured that it's because the community should
have two main sets of developers using Eclipse and IntelliJ, which each tool
has a different way importing formatter and `Organize Imports`/ Import Layout
(Eclipse has to export the `Organize Imports` separately instead of having it
within hbase_eclipse_formatter.xml)
I will send a message to community, if we confirm the import layouts as
proposed, we could add the following the dev-support (at least for the Eclipse)
to align the developer editor.
1. add *.importorder with dev-support for Eclipse
2. (optional) add *.xml for IntelliJ support for formatter and Import Layout
Will see how it goes but at least we have a Jira down for discussion and as a
record for this issue.
was (Author: taklwu):
after I dug into the problem, I figured that it's because the community should
have two main sets of people using Eclipse and IntelliJ, they have different
way for importing formatter and `Organize Imports`/ Import Layout (Eclipse has
to export the `Organize Imports` separately instead of having it within
hbase_eclipse_formatter.xml)
I will send a message to community, if we confirm the import layouts as
proposed, we could add the following the dev-support (at least for the Eclipse)
to align the developer editor.
1. add *.importorder with dev-support for Eclipse
2. (optional) add *.xml for IntelliJ support for formatter and Import Layout
Will see how it goes but at least we have a Jira down for discussion and as a
record for this issue.
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[
https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530597#comment-16530597
]
Hadoop QA commented on HBASE-20836:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-18477 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
51s{color} | {color:red} root in HBASE-18477 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} HBASE-18477 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m
54s{color} | {color:red} branch has 11 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} HBASE-18477 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} hbase-common: The patch generated 1 new + 2 unchanged
- 1 fixed = 3 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 52s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
34s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930024/HBASE-20836.HBASE-18477.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3838813d53df 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-18477 / 2bae94cd73 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HBASE-Build/13473/artifact/patchprocess/branch-mvninstall-root.txt
|
| shadedjars |
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Priority: Minor (was: Major)
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Issue Type: Improvement (was: Task)
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530581#comment-16530581
]
Tak Lon (Stephen) Wu commented on HBASE-20837:
--
after I dug into the problem, I figured that it's because the community should
have two main sets of people using Eclipse and IntelliJ, they have different
way for importing formatter and `Organize Imports`/ Import Layout (Eclipse has
to export the `Organize Imports` separately instead of having it within
hbase_eclipse_formatter.xml)
I will send a message to community, if we confirm the import layouts as
proposed, we could add the following the dev-support (at least for the Eclipse)
to align the developer editor.
1. add *.importorder with dev-support for Eclipse
2. (optional) add *.xml for IntelliJ support for formatter and Import Layout
Will see how it goes but at least we have a Jira down for discussion and as a
record for this issue.
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Description:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
provide a fix here to sync between
[dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
and
[hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
And the proposed import order with the respect to HBASE-19262 and HBASE-19552
should be
!image-2018-07-02-16-33-18-604.png!
was:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between
[dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
and
[hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
And the proposed import order with the respect to HBASE-19262 and HBASE-19552
should be
!image-2018-07-02-15-27-41-140.png!
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !image-2018-07-02-16-33-18-604.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Attachment: image-2018-07-02-16-33-18-604.png
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png,
> image-2018-07-02-16-33-18-604.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530569#comment-16530569 ] Zach York commented on HBASE-20836: --- [~yuzhih...@gmail.com] I'm not sure why the mvninstall and shadedjars is still failing... perhaps it's not applying the patch first because they are failing for the yetus interface audience error. > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch, > HBASE-20836.HBASE-18477.002.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530565#comment-16530565 ] Zach York commented on HBASE-20836: --- Added a new patch to add a private constructor. > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch, > HBASE-20836.HBASE-18477.002.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20836: -- Attachment: HBASE-20836.HBASE-18477.002.patch > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch, > HBASE-20836.HBASE-18477.002.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530546#comment-16530546 ] Ted Yu commented on HBASE-20836: lgtm > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[
https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530521#comment-16530521
]
Hadoop QA commented on HBASE-20836:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-18477 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 3m
39s{color} | {color:red} root in HBASE-18477 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} HBASE-18477 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m
59s{color} | {color:red} branch has 11 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} HBASE-18477 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} hbase-common: The patch generated 1 new + 2 unchanged
- 1 fixed = 3 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 12s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
32s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20836 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930011/HBASE-20836.HBASE-18477.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d52a62fe123d 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-18477 / 2bae94cd73 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HBASE-Build/13472/artifact/patchprocess/branch-mvninstall-root.txt
|
| shadedjars |
[jira] [Commented] (HBASE-20806) Split style journal for flushes and compactions
[ https://issues.apache.org/jira/browse/HBASE-20806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530518#comment-16530518 ] Andrew Purtell commented on HBASE-20806: +1, after patch for master we can commit > Split style journal for flushes and compactions > --- > > Key: HBASE-20806 > URL: https://issues.apache.org/jira/browse/HBASE-20806 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Minor > Attachments: HBASE-20806.branch-1.001.patch > > > In 1.x we have split transaction journal that gives a clear picture of when > various stages of splits took place. We should have a similar thing for > flushes and compactions so as to have insights into time spent in various > stages, which we can use to identify regressions that might creep up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530513#comment-16530513
]
Tak Lon (Stephen) Wu commented on HBASE-20837:
--
I'm working on the IDE part in eclipse format, will update the patch once I
have it. but the importOrder in the checkstyle woule be like this
{code:java}
{code}
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Summary: Sync import orders between IDE and checkstyle module (was: Sync
import orders in checkstyle between IDE and checkstyle module)
> Sync import orders between IDE and checkstyle module
>
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync import orders in checkstyle between IDE and checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Summary: Sync import orders in checkstyle between IDE and checkstyle module
(was: Sync between dev-support/hbase_eclipse_formatter.xml and
hbase/checkstyle.xml)
> Sync import orders in checkstyle between IDE and checkstyle module
> --
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Description:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between
[dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
and
[hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
And the proposed import order with the respect to HBASE-19262 and HBASE-19552
should be
!image-2018-07-02-15-27-41-140.png!
was:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
And the proposed import order with the respect to HBASE-19262 and HBASE-19552
should be
!image-2018-07-02-15-27-41-140.png!
> Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> -
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Attachment: image-2018-07-02-15-27-41-140.png
> Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> -
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Description:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
And the proposed import order with the respect to HBASE-19262 and HBASE-19552
should be
!image-2018-07-02-15-27-41-140.png!
was:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
> Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> -
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Attachment: (was: proposed-import-orders.png)
> Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> -
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: image-2018-07-02-15-27-41-140.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
> !image-2018-07-02-15-27-41-140.png!
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Attachment: proposed-import-orders.png
> Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> -
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
> Attachments: proposed-import-orders.png
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-20837:
-
Description:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting for import order
{code:java}
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
was:
While working on HBASE-20557 contribution, we figured out that the checkstyle
build target (ImportOrder's `groups`
[http://checkstyle.sourceforge.net/config_imports.html] ) was different from
the development supported IDE formatter, we would provide a fix here to sync
between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
This might need to backport the changes of master to branch-1 and branch-2 as
well.
Before this change, this is what checkstyle is expecting
```
import com.google.common.annotations.VisibleForTesting;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.classification.InterfaceAudience;
import org.apache.hadoop.hbase.conf.ConfigurationObserver;
```
> Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> -
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Priority: Major
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE formatter, we would provide a fix here to sync
> between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20817) Infinite loop when executing ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530509#comment-16530509
]
Hudson commented on HBASE-20817:
Results for branch branch-2.1
[build #15 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/15/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/15//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/15//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/15//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Infinite loop when executing ReopenTableRegionsProcedure
> -
>
> Key: HBASE-20817
> URL: https://issues.apache.org/jira/browse/HBASE-20817
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20817-v1.patch, HBASE-20817.patch
>
>
> As discussed in HBASE-20792, it seems that a region's openSeqNum could remain
> the same after a sucessful reopen, which causes the RTRP loop infinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
Tak Lon (Stephen) Wu created HBASE-20837: Summary: Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml Key: HBASE-20837 URL: https://issues.apache.org/jira/browse/HBASE-20837 Project: HBase Issue Type: Task Affects Versions: 1.4.5, 2.0.1, 3.0.0 Reporter: Tak Lon (Stephen) Wu While working on HBASE-20557 contribution, we figured out that the checkstyle build target (ImportOrder's `groups` [http://checkstyle.sourceforge.net/config_imports.html] ) was different from the development supported IDE formatter, we would provide a fix here to sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml This might need to backport the changes of master to branch-1 and branch-2 as well. Before this change, this is what checkstyle is expecting ``` import com.google.common.annotations.VisibleForTesting; import java.io.IOException; import java.util.ArrayList; import java.util.List; import java.util.Map; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.classification.InterfaceAudience; import org.apache.hadoop.hbase.conf.ConfigurationObserver; ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20450) Provide metrics for number of total active, priority and replication rpc handlers
[
https://issues.apache.org/jira/browse/HBASE-20450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530497#comment-16530497
]
Hudson commented on HBASE-20450:
Results for branch branch-2
[build #936 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Provide metrics for number of total active, priority and replication rpc
> handlers
> -
>
> Key: HBASE-20450
> URL: https://issues.apache.org/jira/browse/HBASE-20450
> Project: HBase
> Issue Type: Improvement
> Components: metrics
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0
>
> Attachments: HBASE-20450.master.001.patch,
> HBASE-20450.master.002.patch
>
>
> Currently hbase provides a metric for [number of total active rpc
> handlers|https://github.com/apache/hbase/blob/f4f2b68238a094d7b1931dc8b7939742ccbb2b57/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L187]
> which is a sum of the following:
> * number of active general rpc handlers
> * number of active priority rpc handlers
> * number of active replication rpc handlers
> I think we can have 3 different metrics corresponding to the above mentioned
> handlers which will allow us to see detailed information about number of
> active handlers running for a particular type of handler.
> We can have following new metrics:
> * numActiveGeneralHandler
> * numActivePriorityHandler
> * numActiveReplicationHandler
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20817) Infinite loop when executing ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530496#comment-16530496
]
Hudson commented on HBASE-20817:
Results for branch branch-2
[build #936 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/936//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Infinite loop when executing ReopenTableRegionsProcedure
> -
>
> Key: HBASE-20817
> URL: https://issues.apache.org/jira/browse/HBASE-20817
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20817-v1.patch, HBASE-20817.patch
>
>
> As discussed in HBASE-20792, it seems that a region's openSeqNum could remain
> the same after a sucessful reopen, which causes the RTRP loop infinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20817) Infinite loop when executing ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530495#comment-16530495
]
Hudson commented on HBASE-20817:
Results for branch branch-2.0
[build #502 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/502/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/502//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/502//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/502//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Infinite loop when executing ReopenTableRegionsProcedure
> -
>
> Key: HBASE-20817
> URL: https://issues.apache.org/jira/browse/HBASE-20817
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20817-v1.patch, HBASE-20817.patch
>
>
> As discussed in HBASE-20792, it seems that a region's openSeqNum could remain
> the same after a sucessful reopen, which causes the RTRP loop infinitely.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530488#comment-16530488 ] Zach York commented on HBASE-20836: --- FYI [~te...@apache.org] > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20836: -- Status: Patch Available (was: Open) > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
[ https://issues.apache.org/jira/browse/HBASE-20836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20836: -- Attachment: HBASE-20836.HBASE-18477.001.patch > Add Yetus annotation for ReadReplicaClustersTableNameUtil > - > > Key: HBASE-20836 > URL: https://issues.apache.org/jira/browse/HBASE-20836 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: HBASE-18477 > > Attachments: HBASE-20836.HBASE-18477.001.patch > > > Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20836) Add Yetus annotation for ReadReplicaClustersTableNameUtil
Zach York created HBASE-20836: - Summary: Add Yetus annotation for ReadReplicaClustersTableNameUtil Key: HBASE-20836 URL: https://issues.apache.org/jira/browse/HBASE-20836 Project: HBase Issue Type: Sub-task Affects Versions: HBASE-18477 Reporter: Zach York Assignee: Zach York Fix For: HBASE-18477 Found via nightly builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20813) Remove RPC quotas when the associated table/Namespace is dropped off
[
https://issues.apache.org/jira/browse/HBASE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20813:
---
Summary: Remove RPC quotas when the associated table/Namespace is dropped
off (was: Remove RPC quotas when the associated table is dropped off)
> Remove RPC quotas when the associated table/Namespace is dropped off
>
>
> Key: HBASE-20813
> URL: https://issues.apache.org/jira/browse/HBASE-20813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
>
> In short, the below scenario shouldn't be the case.
> {noformat}
> hbase(main):023:0> create 't2','cf1'
> Created table t2
> Took 0.7405 seconds
> => Hbase::Table - t2
> hbase(main):024:0>
> hbase(main):025:0*
> hbase(main):026:0* set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0082 seconds
> hbase(main):027:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0291 seconds
> hbase(main):028:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0037 seconds
> hbase(main):029:0> disable 't2'
> Took 0.4328 seconds
> hbase(main):030:0> drop 't2'
> Took 0.2285 seconds
> hbase(main):031:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0230 seconds
> hbase(main):032:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0038 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20813) Remove RPC quotas when the associated table is dropped off
[
https://issues.apache.org/jira/browse/HBASE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530444#comment-16530444
]
Sakthi commented on HBASE-20813:
[~mdrob], [~elserj]. I have question. I see that we have a class
"MasterSpaceQuotaObserver" that is "observing" the space quota management. From
here, we could go 2 ways. Either creating a new class naming
"MasterThrottleQuotaObserver" or renaming "MasterSpaceQuotaObserver" into
"MasterQuotasObserver" and managing in a single class.
I feel, having a separate "observer"(MasterThrottleQuotaObserver) would be more
fruitful for us, in case, if we want to add exclusive properties to any
specific "observers". What do you guys think?
Note: All of this, while using the same config property.
> Remove RPC quotas when the associated table is dropped off
> --
>
> Key: HBASE-20813
> URL: https://issues.apache.org/jira/browse/HBASE-20813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
>
> In short, the below scenario shouldn't be the case.
> {noformat}
> hbase(main):023:0> create 't2','cf1'
> Created table t2
> Took 0.7405 seconds
> => Hbase::Table - t2
> hbase(main):024:0>
> hbase(main):025:0*
> hbase(main):026:0* set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0082 seconds
> hbase(main):027:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0291 seconds
> hbase(main):028:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0037 seconds
> hbase(main):029:0> disable 't2'
> Took 0.4328 seconds
> hbase(main):030:0> drop 't2'
> Took 0.2285 seconds
> hbase(main):031:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0230 seconds
> hbase(main):032:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0038 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20813) Remove RPC quotas when the associated table is dropped off
[
https://issues.apache.org/jira/browse/HBASE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530444#comment-16530444
]
Sakthi edited comment on HBASE-20813 at 7/2/18 9:12 PM:
[~mdrob], [~elserj]. I have question. I see that we have a class
"MasterSpaceQuotaObserver" that is "observing" the space quota automatic
deletion. From here, we could go 2 ways. Either creating a new class naming
"MasterThrottleQuotaObserver" or renaming "MasterSpaceQuotaObserver" into
"MasterQuotasObserver" and managing in a single class.
I feel, having a separate "observer"(MasterThrottleQuotaObserver) would be more
fruitful for us, in case, if we want to add exclusive properties to any
specific "observers". What do you guys think?
Note: All of this, while using the same config property.
was (Author: jatsakthi):
[~mdrob], [~elserj]. I have question. I see that we have a class
"MasterSpaceQuotaObserver" that is "observing" the space quota management. From
here, we could go 2 ways. Either creating a new class naming
"MasterThrottleQuotaObserver" or renaming "MasterSpaceQuotaObserver" into
"MasterQuotasObserver" and managing in a single class.
I feel, having a separate "observer"(MasterThrottleQuotaObserver) would be more
fruitful for us, in case, if we want to add exclusive properties to any
specific "observers". What do you guys think?
Note: All of this, while using the same config property.
> Remove RPC quotas when the associated table is dropped off
> --
>
> Key: HBASE-20813
> URL: https://issues.apache.org/jira/browse/HBASE-20813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
>
> In short, the below scenario shouldn't be the case.
> {noformat}
> hbase(main):023:0> create 't2','cf1'
> Created table t2
> Took 0.7405 seconds
> => Hbase::Table - t2
> hbase(main):024:0>
> hbase(main):025:0*
> hbase(main):026:0* set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0082 seconds
> hbase(main):027:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0291 seconds
> hbase(main):028:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0037 seconds
> hbase(main):029:0> disable 't2'
> Took 0.4328 seconds
> hbase(main):030:0> drop 't2'
> Took 0.2285 seconds
> hbase(main):031:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0230 seconds
> hbase(main):032:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0038 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[ https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530432#comment-16530432 ] Zach York commented on HBASE-18477: --- [~yuzhih...@gmail.com] Thanks for pointing out. I will address that. > Umbrella JIRA for HBase Read Replica clusters > - > > Key: HBASE-18477 > URL: https://issues.apache.org/jira/browse/HBASE-18477 > Project: HBase > Issue Type: New Feature >Reporter: Zach York >Assignee: Zach York >Priority: Major > Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase > Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope > doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf > > > Recently, changes (such as HBASE-17437) have unblocked HBase to run with a > root directory external to the cluster (such as in Amazon S3). This means > that the data is stored outside of the cluster and can be accessible after > the cluster has been terminated. One use case that is often asked about is > pointing multiple clusters to one root directory (sharing the data) to have > read resiliency in the case of a cluster failure. > > This JIRA is an umbrella JIRA to contain all the tasks necessary to create a > read-replica HBase cluster that is pointed at the same root directory. > > This requires making the Read-Replica cluster Read-Only (no metadata > operation or data operations). > Separating the hbase:meta table for each cluster (Otherwise HBase gets > confused with multiple clusters trying to update the meta table with their ip > addresses) > Adding refresh functionality for the meta table to ensure new metadata is > picked up on the read replica cluster. > Adding refresh functionality for HFiles for a given table to ensure new data > is picked up on the read replica cluster. > > This can be used with any existing cluster that is backed by an external > filesystem. > > Please note that this feature is still quite manual (with the potential for > automation later). > > More information on this particular feature can be found here: > https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20450) Provide metrics for number of total active, priority and replication rpc handlers
[ https://issues.apache.org/jira/browse/HBASE-20450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-20450. Resolution: Fixed > Provide metrics for number of total active, priority and replication rpc > handlers > - > > Key: HBASE-20450 > URL: https://issues.apache.org/jira/browse/HBASE-20450 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-20450.master.001.patch, > HBASE-20450.master.002.patch > > > Currently hbase provides a metric for [number of total active rpc > handlers|https://github.com/apache/hbase/blob/f4f2b68238a094d7b1931dc8b7939742ccbb2b57/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L187] > which is a sum of the following: > * number of active general rpc handlers > * number of active priority rpc handlers > * number of active replication rpc handlers > I think we can have 3 different metrics corresponding to the above mentioned > handlers which will allow us to see detailed information about number of > active handlers running for a particular type of handler. > We can have following new metrics: > * numActiveGeneralHandler > * numActivePriorityHandler > * numActiveReplicationHandler > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530378#comment-16530378 ] Mohit Goel commented on HBASE-6028: --- Thanks [~yuzhih...@gmail.com], will take care in the future. > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > Attachments: HBASE-6028.master.007.patch, HBASE-6028.master.008.patch > > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Goel updated HBASE-6028: -- Attachment: HBASE-6028.master.008.patch > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > Attachments: HBASE-6028.master.007.patch, HBASE-6028.master.008.patch > > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16110) AsyncFS WAL doesn't work with Hadoop 2.8+
[ https://issues.apache.org/jira/browse/HBASE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530352#comment-16530352 ] Sean Busbey commented on HBASE-16110: - Hi [~timoha]! Please bring issues running HBase to the user@hbase mailing list. JIRA is for tracking specific work tasks and this one is unlikely to get responses since it was resolved a couple of years ago. > AsyncFS WAL doesn't work with Hadoop 2.8+ > - > > Key: HBASE-16110 > URL: https://issues.apache.org/jira/browse/HBASE-16110 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16110-v1.patch, HBASE-16110.patch > > > The async wal implementation doesn't work with Hadoop 2.8+. Fails compilation > and will fail running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-16110) AsyncFS WAL doesn't work with Hadoop 2.8+
[
https://issues.apache.org/jira/browse/HBASE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530323#comment-16530323
]
Sean Busbey edited comment on HBASE-16110 at 7/2/18 7:39 PM:
-
Hello, we are running HBase 2.0.1 with official Hadoop 2.8.4 jars and hadoop
2.8.4 client
([http://central.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.4/]). Got
the following exception on regionserver which brings it down:
{code}
18/07/02 18:51:06 WARN concurrent.DefaultPromise: An exception was thrown by
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete()
java.lang.Error: Couldn't properly initialize access to HDFS internals. Please
update your WAL Provider to not make use of the 'asyncfs' provider. See
HBASE-16110 for more information.
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:268)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)
at java.lang.Class.getDeclaredMethod(Class.java:2130)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)
... 18 more
{code}
FYI, we don't have encryption enabled. Let me know if you need more info about
our setup.
was (Author: timoha):
Hello, we are running HBase 2.0.1 with official Hadoop 2.8.4 jars and hadoop
2.8.4 client
([http://central.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.4/]). Got
the following exception on regionserver which brings it down:
{{ 18/07/02 18:51:06 WARN concurrent.DefaultPromise: An exception was thrown by
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete()}}
{{ java.lang.Error: Couldn't properly initialize access to HDFS internals.
Please update your WAL Provider to not make use of the 'asyncfs' provider. See
HBASE-16110 for more information.}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:268)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)}}
{{ at
[jira] [Comment Edited] (HBASE-16110) AsyncFS WAL doesn't work with Hadoop 2.8+
[
https://issues.apache.org/jira/browse/HBASE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530323#comment-16530323
]
Andrey Elenskiy edited comment on HBASE-16110 at 7/2/18 7:14 PM:
-
Hello, we are running HBase 2.0.1 with official Hadoop 2.8.4 jars and hadoop
2.8.4 client
([http://central.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.4/]). Got
the following exception on regionserver which brings it down:
{{ 18/07/02 18:51:06 WARN concurrent.DefaultPromise: An exception was thrown by
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete()}}
{{ java.lang.Error: Couldn't properly initialize access to HDFS internals.
Please update your WAL Provider to not make use of the 'asyncfs' provider. See
HBASE-16110 for more information.}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:268)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)}}
{{ at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)}}
{{ at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638)}}
{{ at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676)}}
{{ at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552)}}
{{ at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)}}
{{ at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)}}
{{ at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{ Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo)}}
{{ at java.lang.Class.getDeclaredMethod(Class.java:2130)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232)}}
{{ at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262)}}
{{ ... 18 more}}
FYI, we don't have encryption enabled. Let me know if you need more info about
our setup.
was (Author: timoha):
Hello, we are running HBase 2.0.1 with official Hadoop 2.8.4 jars and hadoop
2.8.4 client
(http://central.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.4/). Got
the following exception on regionserver which brings it down:
```
18/07/02 18:51:06 WARN concurrent.DefaultPromise: An exception was thrown by
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete()
java.lang.Error: Couldn't properly initialize access to HDFS internals. Please
update your WAL Provider to not make use of the 'asyncfs' provider. See
HBASE-16110 for more information.
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:268)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661)
at
[jira] [Commented] (HBASE-16110) AsyncFS WAL doesn't work with Hadoop 2.8+
[ https://issues.apache.org/jira/browse/HBASE-16110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530323#comment-16530323 ] Andrey Elenskiy commented on HBASE-16110: - Hello, we are running HBase 2.0.1 with official Hadoop 2.8.4 jars and hadoop 2.8.4 client (http://central.maven.org/maven2/org/apache/hadoop/hadoop-client/2.8.4/). Got the following exception on regionserver which brings it down: ``` 18/07/02 18:51:06 WARN concurrent.DefaultPromise: An exception was thrown by org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete() java.lang.Error: Couldn't properly initialize access to HDFS internals. Please update your WAL Provider to not make use of the 'asyncfs' provider. See HBASE-16110 for more information. at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:268) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:232) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:262) ... 18 more ``` FYI, we don't have encryption enabled. > AsyncFS WAL doesn't work with Hadoop 2.8+ > - > > Key: HBASE-16110 > URL: https://issues.apache.org/jira/browse/HBASE-16110 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16110-v1.patch, HBASE-16110.patch > > > The async wal implementation doesn't work with Hadoop 2.8+. Fails compilation > and will fail running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20450) Provide metrics for number of total active, priority and replication rpc handlers
[ https://issues.apache.org/jira/browse/HBASE-20450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20450: --- Fix Version/s: 1.5.0 2.1.0 > Provide metrics for number of total active, priority and replication rpc > handlers > - > > Key: HBASE-20450 > URL: https://issues.apache.org/jira/browse/HBASE-20450 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Nihal Jain >Assignee: Nihal Jain >Priority: Major > Fix For: 3.0.0, 2.1.0, 1.5.0 > > Attachments: HBASE-20450.master.001.patch, > HBASE-20450.master.002.patch > > > Currently hbase provides a metric for [number of total active rpc > handlers|https://github.com/apache/hbase/blob/f4f2b68238a094d7b1931dc8b7939742ccbb2b57/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L187] > which is a sum of the following: > * number of active general rpc handlers > * number of active priority rpc handlers > * number of active replication rpc handlers > I think we can have 3 different metrics corresponding to the above mentioned > handlers which will allow us to see detailed information about number of > active handlers running for a particular type of handler. > We can have following new metrics: > * numActiveGeneralHandler > * numActivePriorityHandler > * numActiveReplicationHandler > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530205#comment-16530205 ] Mike Drob commented on HBASE-20781: --- bq. Pushed to branch-2.0+. Thanks for review [~Apache9]. I only see this on branch-2 and master, not -2.0 or -2.1 -- need additional backports here? > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.branch-2.0.002.patch, HBASE-20781.branch-2.0.003.patch, > HBASE-20781.master.001.patch, hitsfamily.png, hitsfamilyagain.png > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20781: -- Fix Version/s: 2.2.0 3.0.0 > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.branch-2.0.002.patch, HBASE-20781.branch-2.0.003.patch, > HBASE-20781.master.001.patch, hitsfamily.png, hitsfamilyagain.png > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20791) RSGroupBasedLoadBalancer#setClusterMetrics should pass ClusterMetrics to its internalBalancer
[ https://issues.apache.org/jira/browse/HBASE-20791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530167#comment-16530167 ] Andrew Purtell commented on HBASE-20791: RSGroups is a feature in branch-1 too. Does this issue also affect branch-1? > RSGroupBasedLoadBalancer#setClusterMetrics should pass ClusterMetrics to its > internalBalancer > - > > Key: HBASE-20791 > URL: https://issues.apache.org/jira/browse/HBASE-20791 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 3.0.0, 2.0.0 >Reporter: chenxu >Assignee: chenxu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 20791-master-v2.patch, HBASE-20791-branch-2-v1.patch, > HBASE-20791-master-v1.patch, HBASE-20791-master-v3.patch, > HBASE-20791-master-v4.patch > > > RSGroupBasedLoadBalancer#setClusterMetrics should pass ClusterMetrics to it’s > internalBalancer, Or the StochasticLoadBalancer(internal balancer) will lose > it's Up-to-date RegionLoads info, and effect the balance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530109#comment-16530109
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #252 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/252/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/252//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/252//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/252//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/252//console].
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20829) TestSyncReplicationStandbyKillRS is flakey
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530069#comment-16530069
]
Hadoop QA commented on HBASE-20829:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} hbase-server: The patch generated 0 new + 6
unchanged - 1 fixed = 6 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 52s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 4s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterProcedureScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20829 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929958/HBASE-20829.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 8ffc5a452026 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / cfdabe9267 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/13470/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Updated] (HBASE-20835) Document how to get replication reporting
[ https://issues.apache.org/jira/browse/HBASE-20835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20835: -- Priority: Critical (was: Major) > Document how to get replication reporting > - > > Key: HBASE-20835 > URL: https://issues.apache.org/jira/browse/HBASE-20835 > Project: HBase > Issue Type: Task > Components: Replication >Affects Versions: 2.1.0 >Reporter: Mike Drob >Assignee: Duo Zhang >Priority: Critical > Fix For: 2.1.0 > > > Based on my questions at the tail end of HBASE-19543 > bq. We have some tooling that checks on replication queues and reads the > znode as the source of truth. When replication is disabled, it's expected > that the node was still there, but just empty. Is there a better way to get > this same information? > I understand that with table based replication it doesn't make sense to check > ZK for status. However, losing the ability to inspect the data and get > information is a tough hit for operators. Do we have APIs that expose the > same sort of metrics? > bq. how many peers/queues, queue size, position in the queue, and age of last > op > Assigning to you for now, Duo, since you were both primary implementor and RM > for 2.1.0 and I'm not sure who else would know the answers. If the docs > already exist, then nothing to do but we should include them in the RN. Maybe > this will need additional code, but I hope it's already there and is > something we can write a workaround for. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20835) Document how to get replication reporting
Mike Drob created HBASE-20835: - Summary: Document how to get replication reporting Key: HBASE-20835 URL: https://issues.apache.org/jira/browse/HBASE-20835 Project: HBase Issue Type: Task Components: Replication Affects Versions: 2.1.0 Reporter: Mike Drob Assignee: Duo Zhang Fix For: 2.1.0 Based on my questions at the tail end of HBASE-19543 bq. We have some tooling that checks on replication queues and reads the znode as the source of truth. When replication is disabled, it's expected that the node was still there, but just empty. Is there a better way to get this same information? I understand that with table based replication it doesn't make sense to check ZK for status. However, losing the ability to inspect the data and get information is a tough hit for operators. Do we have APIs that expose the same sort of metrics? bq. how many peers/queues, queue size, position in the queue, and age of last op Assigning to you for now, Duo, since you were both primary implementor and RM for 2.1.0 and I'm not sure who else would know the answers. If the docs already exist, then nothing to do but we should include them in the RN. Maybe this will need additional code, but I hope it's already there and is something we can write a workaround for. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20834) The jenkins on http://104.198.223.121:8080/job/HBASE-Flaky-Tests/ is broken
[ https://issues.apache.org/jira/browse/HBASE-20834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20834: Component/s: test community > The jenkins on http://104.198.223.121:8080/job/HBASE-Flaky-Tests/ is broken > --- > > Key: HBASE-20834 > URL: https://issues.apache.org/jira/browse/HBASE-20834 > Project: HBase > Issue Type: Bug > Components: community, test >Reporter: Duo Zhang >Priority: Major > > It is used by our flakey test finder to collect flakey tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[ https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529996#comment-16529996 ] Sean Busbey commented on HBASE-20331: - only the docs left to do, but I'm caught up in a $dayjob thing at the moment. fine to bump out the resolution, IMO. easy enough to add docs in a 2.1.1 release. > clean up shaded packaging for 2.1 > - > > Key: HBASE-20331 > URL: https://issues.apache.org/jira/browse/HBASE-20331 > Project: HBase > Issue Type: Umbrella > Components: Client, mapreduce, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 3.0.0, 2.2.0 > > > polishing pass on shaded modules for 2.0 based on trying to use them in more > contexts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20829) TestSyncReplicationStandbyKillRS is flakey
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529985#comment-16529985
]
Duo Zhang commented on HBASE-20829:
---
The problem is that for MasterProcedureScheduler.yield, the addFront parameter
will be set to false if the proc is not a TableProcedure. But in doAdd method,
if the parent proc has the lock, then it asserts that the addFront parameter
must be true. I think this is a general problem, should be fixed at least for
branch-2.1+. If the pre commit result is OK, will change the title of this
issue to better describe the change, and prepare the patch for branches other
than master.
> TestSyncReplicationStandbyKillRS is flakey
> --
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20829) TestSyncReplicationStandbyKillRS is flakey
[
https://issues.apache.org/jira/browse/HBASE-20829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-20829:
--
Status: Patch Available (was: Open)
> TestSyncReplicationStandbyKillRS is flakey
> --
>
> Key: HBASE-20829
> URL: https://issues.apache.org/jira/browse/HBASE-20829
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20829-debug.patch, HBASE-20829.patch,
> org.apache.hadoop.hbase.replication.TestSyncReplicationStandbyKillRS-output.txt
>
>
> Timed out.
> {noformat}
> 2018-06-30 01:32:33,823 ERROR [Time-limited test]
> replication.TestSyncReplicationStandbyKillRS(93): Failed to transit standby
> cluster to DOWNGRADE_ACTIVE
> {noformat}
> We failed to transit the state to DA and then wait for it to become DA so
> hang there.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
