[jira] [Commented] (HBASE-21018) RS crashed because AsyncFS was unable to update HDFS data encryption key
[
https://issues.apache.org/jira/browse/HBASE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571147#comment-16571147
]
Duo Zhang commented on HBASE-21018:
---
+1.
> RS crashed because AsyncFS was unable to update HDFS data encryption key
>
>
> Key: HBASE-21018
> URL: https://issues.apache.org/jira/browse/HBASE-21018
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0, HBase 2.0.0,
> HDFS configuration dfs.encrypt.data.transfer = true
>Reporter: Wei-Chiu Chuang
>Priority: Critical
> Attachments: HBASE-21018.master.001.patch
>
>
> We (+[~uagashe]) found HBase RegionServer doesn't update HDFS data encryption
> key correctly, and in some cases after retry 10 times, it aborts.
> {noformat}
> 2018-08-03 17:37:03,233 WARN
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create
> fan-out dfs output
> /hbase/WALs/rs1.example.com,22101,1533318719239/rs1.example.com%2C22101%2C1533318719239.rs1.example.com%2C22101%2C1533318719239.regiongroup-0.1533343022981
> failed, retry = 1
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=1685436998) doesn't exist. Current key: 1085959374
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.check(FanOutOneBlockAsyncDFSOutputSaslHelper.java:399)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.channelRead(FanOutOneBlockAsyncDFSOutputSaslHelper.java:470)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
> at
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
> at
>
[jira] [Commented] (HBASE-20972) Fix call queue buffer size leaking bug
[ https://issues.apache.org/jira/browse/HBASE-20972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571142#comment-16571142 ] Xiaolin Ha commented on HBASE-20972: [~yuzhih...@gmail.com] Call queue size is the currently queued and running Calls bytes size. This method is to subtract call queue size when a call is failed or rejected. So shouldn't we remove the word 'Queue' ? > Fix call queue buffer size leaking bug > -- > > Key: HBASE-20972 > URL: https://issues.apache.org/jira/browse/HBASE-20972 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Affects Versions: 2.1.0, 2.0.0, 2.2.0 >Reporter: Xiaolin Ha >Assignee: Xiaolin Ha >Priority: Major > Attachments: HBASE-20972.branch-2.0.001.patch > > > Call queue size is the currently queued and running Calls bytes size. It gets > incremented after we parse a call and before we add it to the queue of calls > for the scheduler to use. It get decremented after we have 'run' the Call. > When setting up a call, total size of it is added. So when a new call can not > be dispatched by BlockingQueue full, the call queue size should be > decremented. We shouldn't add size of rejected calls to the call queue size. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20986) Separate the config of block size when we do log splitting and write Hlog
[
https://issues.apache.org/jira/browse/HBASE-20986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571132#comment-16571132
]
Hadoop QA commented on HBASE-20986:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
53s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
57s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-20986 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934585/HBASE-20986.branch-2.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0af8b60e361d 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2 / b2cefab0e1 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13946/testReport/ |
| Max. process+thread count | 4453 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-20965) Separate region server report requests to new handlers
[ https://issues.apache.org/jira/browse/HBASE-20965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571129#comment-16571129 ] Yu Li commented on HBASE-20965: --- bq. So I thought we can optimizate it in a new issue, both SimpleRpcScheduler and this new MasterFifoRpcScheduleer. Sounds good, expecting the new JIRA and a link here then. Please also add a release note here to indicate the change, that we will have dedicated (configurable) handler count for RS report in HMaster. Thanks. > Separate region server report requests to new handlers > -- > > Key: HBASE-20965 > URL: https://issues.apache.org/jira/browse/HBASE-20965 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20965.master.001.patch, > HBASE-20965.master.002.patch, HBASE-20965.master.003.patch, > HBASE-20965.master.004.patch, HBASE-20965.master.005.patch, > HBASE-20965.master.006.patch, HBASE-20965.master.007.patch > > > In master rpc scheduler, all rpc requests are executed in a thread pool. This > task separates rs report requests to new handlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan-Po Tseng updated HBASE-18201: -- Attachment: HBASE-18201.master.006.patch > add UT and docs for DataBlockEncodingTool > - > > Key: HBASE-18201 > URL: https://issues.apache.org/jira/browse/HBASE-18201 > Project: HBase > Issue Type: Sub-task > Components: tooling >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng >Priority: Minor > Labels: beginner > Attachments: HBASE-18201.master.001.patch, > HBASE-18201.master.002.patch, HBASE-18201.master.002.patch, > HBASE-18201.master.003.patch, HBASE-18201.master.004.patch, > HBASE-18201.master.005.patch, HBASE-18201.master.005.patch, > HBASE-18201.master.005.patch, HBASE-18201.master.006.patch > > > There is no example, documents, or tests for DataBlockEncodingTool. We should > have it friendly if any use case exists. Otherwise, we should just get rid of > it because DataBlockEncodingTool presumes that the implementation of cell > returned from DataBlockEncoder is KeyValue. The presume may obstruct the > cleanup of KeyValue references in the code base of read/write path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571119#comment-16571119
]
Jack Bearden commented on HBASE-20993:
--
It's okay, I got it. Just need a little more time
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
> org.playground.hbase.KerberizedClientFallback.main(KerberizedClientFallback.java:55)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
>
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571117#comment-16571117
]
Reid Chan commented on HBASE-20993:
---
Feel free to release it if you don't have enough cycles, just saw you dive into
many issues.
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
> org.playground.hbase.KerberizedClientFallback.main(KerberizedClientFallback.java:55)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
>
[jira] [Updated] (HBASE-20965) Separate region server report requests to new handlers
[ https://issues.apache.org/jira/browse/HBASE-20965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-20965: --- Attachment: HBASE-20965.master.007.patch > Separate region server report requests to new handlers > -- > > Key: HBASE-20965 > URL: https://issues.apache.org/jira/browse/HBASE-20965 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20965.master.001.patch, > HBASE-20965.master.002.patch, HBASE-20965.master.003.patch, > HBASE-20965.master.004.patch, HBASE-20965.master.005.patch, > HBASE-20965.master.006.patch, HBASE-20965.master.007.patch > > > In master rpc scheduler, all rpc requests are executed in a thread pool. This > task separates rs report requests to new handlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20965) Separate region server report requests to new handlers
[ https://issues.apache.org/jira/browse/HBASE-20965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-20965: --- Attachment: HBASE-20965.master.006.patch > Separate region server report requests to new handlers > -- > > Key: HBASE-20965 > URL: https://issues.apache.org/jira/browse/HBASE-20965 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20965.master.001.patch, > HBASE-20965.master.002.patch, HBASE-20965.master.003.patch, > HBASE-20965.master.004.patch, HBASE-20965.master.005.patch, > HBASE-20965.master.006.patch > > > In master rpc scheduler, all rpc requests are executed in a thread pool. This > task separates rs report requests to new handlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20957) Switch replication over to use the new WAL API
[ https://issues.apache.org/jira/browse/HBASE-20957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571109#comment-16571109 ] Anoop Sam John commented on HBASE-20957: We dont have the WAL new API system yet right? > Switch replication over to use the new WAL API > -- > > Key: HBASE-20957 > URL: https://issues.apache.org/jira/browse/HBASE-20957 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Ankit Singhal >Priority: Major > > Replication needs to use the new WAL APIs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571108#comment-16571108
]
Jack Bearden commented on HBASE-20993:
--
Awesome, that is helpful. I'll make more cycles on this and report back
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
> org.playground.hbase.KerberizedClientFallback.main(KerberizedClientFallback.java:55)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
> at
>
[jira] [Updated] (HBASE-20965) Separate region server report requests to new handlers
[ https://issues.apache.org/jira/browse/HBASE-20965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Mei updated HBASE-20965: --- Attachment: HBASE-20965.master.005.patch > Separate region server report requests to new handlers > -- > > Key: HBASE-20965 > URL: https://issues.apache.org/jira/browse/HBASE-20965 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20965.master.001.patch, > HBASE-20965.master.002.patch, HBASE-20965.master.003.patch, > HBASE-20965.master.004.patch, HBASE-20965.master.005.patch > > > In master rpc scheduler, all rpc requests are executed in a thread pool. This > task separates rs report requests to new handlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571105#comment-16571105
]
Reid Chan commented on HBASE-20993:
---
My simple client code (snippet), fyi.
{code}
final Configuration conf = HBaseConfiguration.create();
conf.addResource("krb-hbase-site.xml");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("whoever@REALM", keytabLocation);
UserGroupInformation user = UserGroupInformation.getLoginUser();
TableName tableName = TableName.valueOf("client_fall_back");
HTableDescriptor desc = new HTableDescriptor(tableName);
String family = "cfb";
HColumnDescriptor columnDescriptor = new HColumnDescriptor(family);
columnDescriptor.setInMemory(true);
desc.addFamily(columnDescriptor);
Connection connection = null;
Admin admin = null;
try {
connection = user.doAs(new PrivilegedExceptionAction() {
@Override
public Connection run() throws Exception {
return ConnectionFactory.createConnection(conf);
}
});
admin = connection.getAdmin();
if (admin.tableExists(tableName)) {
admin.disableTable(tableName);
admin.deleteTable(tableName);
}
admin.createTable(desc);
System.out.println("Table created!");
} catch (InterruptedException e) {
System.err.println("Interrupted while creating connection.");
System.exit(1);
} finally {
if (connection != null) {
connection.close();
}
if (admin != null) {
admin.close();
}
}
{code}
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
>
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571104#comment-16571104
]
Reid Chan commented on HBASE-20993:
---
bq. I have been writing my own client but haven't been able to get that error
yet
Does your client kerberized?
bq. Are there any existing tests I could also use as a framework to debug this?
No, i wrote my simple client too.
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:607)
> at
> org.playground.hbase.KerberizedClientFallback.main(KerberizedClientFallback.java:55)
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
> at
>
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571098#comment-16571098
]
Jack Bearden commented on HBASE-20993:
--
Hi [~reidchan]! Thank you for providing guidance on this bug. I have been
working to reproduce it but do not yet see what you got. I am a bit stuck so I
have a few questions for you :)
# I was curious what client you used to get the errors there? Knowing this I
could setup a debugger and suspend around where the error is thrown. I have
been writing my own client but haven't been able to get that error yet
# Are there any existing tests I could also use as a framework to debug this?
I looked extensively and wasn't able to find any obvious use-cases. This would
be ideal if you knew a test that had a similar composition I can start from.
The RPC nature of this is a bit tricky... maybe mockito?
>From what I can tell reading the SASL RFC#, the negotiate phase appears to
>be what they recommend when dealing with multiple security layers. I do agree
>that that approach is a solid way to go. Thanks again in advance!
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, security
>Affects Versions: 1.2.6
>Reporter: Reid Chan
>Assignee: Jack Bearden
>Priority: Critical
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
>
[jira] [Commented] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[
https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571096#comment-16571096
]
Hadoop QA commented on HBASE-20881:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} hbase-client: The patch generated 0 new + 2
unchanged - 84 fixed = 2 total (was 86) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} hbase-procedure: The patch generated 0 new + 20
unchanged - 1 fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} hbase-server: The patch generated 0 new + 253
unchanged - 49 fixed = 253 total (was 302) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 43s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
13s{color} | {color:green} hbase-client in the patch passed. {color} |
|
[jira] [Commented] (HBASE-21018) RS crashed because AsyncFS was unable to update HDFS data encryption key
[
https://issues.apache.org/jira/browse/HBASE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571064#comment-16571064
]
Umesh Agashe commented on HBASE-21018:
--
Hi [~Apache9], any details you can provide on this will be helpful. thanks!
> RS crashed because AsyncFS was unable to update HDFS data encryption key
>
>
> Key: HBASE-21018
> URL: https://issues.apache.org/jira/browse/HBASE-21018
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0, HBase 2.0.0,
> HDFS configuration dfs.encrypt.data.transfer = true
>Reporter: Wei-Chiu Chuang
>Priority: Critical
> Attachments: HBASE-21018.master.001.patch
>
>
> We (+[~uagashe]) found HBase RegionServer doesn't update HDFS data encryption
> key correctly, and in some cases after retry 10 times, it aborts.
> {noformat}
> 2018-08-03 17:37:03,233 WARN
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create
> fan-out dfs output
> /hbase/WALs/rs1.example.com,22101,1533318719239/rs1.example.com%2C22101%2C1533318719239.rs1.example.com%2C22101%2C1533318719239.regiongroup-0.1533343022981
> failed, retry = 1
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=1685436998) doesn't exist. Current key: 1085959374
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.check(FanOutOneBlockAsyncDFSOutputSaslHelper.java:399)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.channelRead(FanOutOneBlockAsyncDFSOutputSaslHelper.java:470)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
> at
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
>
[jira] [Commented] (HBASE-20986) Separate the config of block size when we do log splitting and write Hlog
[ https://issues.apache.org/jira/browse/HBASE-20986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571063#comment-16571063 ] Jingyun Tian commented on HBASE-20986: -- Patch for branch-2 is uploaded. Thanks. > Separate the config of block size when we do log splitting and write Hlog > - > > Key: HBASE-20986 > URL: https://issues.apache.org/jira/browse/HBASE-20986 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.1.0, 2.0.1, 2.2.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20986.branch-2.001.patch, > HBASE-20986.master.001.patch, HBASE-20986.master.002.patch, > HBASE-20986.master.003.patch, HBASE-20986.master.004.patch > > > Since the block size of recovered edits and hlog are the same right now, if > we set a large value to block size, name node may not able to assign enough > space when we do log splitting. But set a large value to hlog block size can > help reduce the number of region server asking for a new block. Thus I think > separate the config of block size is necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20986) Separate the config of block size when we do log splitting and write Hlog
[ https://issues.apache.org/jira/browse/HBASE-20986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-20986: - Attachment: HBASE-20986.branch-2.001.patch > Separate the config of block size when we do log splitting and write Hlog > - > > Key: HBASE-20986 > URL: https://issues.apache.org/jira/browse/HBASE-20986 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.1.0, 2.0.1, 2.2.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20986.branch-2.001.patch, > HBASE-20986.master.001.patch, HBASE-20986.master.002.patch, > HBASE-20986.master.003.patch, HBASE-20986.master.004.patch > > > Since the block size of recovered edits and hlog are the same right now, if > we set a large value to block size, name node may not able to assign enough > space when we do log splitting. But set a large value to hlog block size can > help reduce the number of region server asking for a new block. Thus I think > separate the config of block size is necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20985) add two attributes when we do normalization
[
https://issues.apache.org/jira/browse/HBASE-20985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571061#comment-16571061
]
Guanghao Zhang commented on HBASE-20985:
get/set should use same name pattern And the static config should use same
name too.
{code:java}
296 public int getNormalizerTargetRegionCount() {
297 return
getDelegateeForModification().getNormalizerTargetRegionCount();
298 }
299
300 public HTableDescriptor setNormalizationTargetRegionCount(final int
regionCount) {
301
getDelegateeForModification().setNormalizationTargetRegionCount(regionCount);
302 return this;
303 }
304
305 @Override
306 public long getNormalizerTargetRegionSize() {
307 return
getDelegateeForModification().getNormalizerTargetRegionSize();
308 }
309
310 public HTableDescriptor setNormalizationTargetRegionSize(final long
regionSize) {
311
getDelegateeForModification().setNormalizationTargetRegionSize(regionSize);
312 return this;
313 }
{code}
> add two attributes when we do normalization
> ---
>
> Key: HBASE-20985
> URL: https://issues.apache.org/jira/browse/HBASE-20985
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HBASE-20985.master.001.patch,
> HBASE-20985.master.002.patch, HBASE-20985.master.003.patch,
> HBASE-20985.master.004.patch
>
>
> Currently when we turn on normalization switch, it will help balance the
> whole table based on total region size / total region count. I add two
> attributes so that we can set total region count or average region size we
> want to achieve when normalization done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20965) Separate region server report requests to new handlers
[ https://issues.apache.org/jira/browse/HBASE-20965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571057#comment-16571057 ] Guanghao Zhang commented on HBASE-20965: bq. I think this is a good question and I'd suggest to use StealJobQueue to better utilize handlers. Now SimpleRpcScheduer use config to decide how many handlers to read/write/replication, too. So I thought we can optimizate it in a new issue, both SimpleRpcScheduler and this new MasterFifoRpcScheduleer. > Separate region server report requests to new handlers > -- > > Key: HBASE-20965 > URL: https://issues.apache.org/jira/browse/HBASE-20965 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20965.master.001.patch, > HBASE-20965.master.002.patch, HBASE-20965.master.003.patch, > HBASE-20965.master.004.patch > > > In master rpc scheduler, all rpc requests are executed in a thread pool. This > task separates rs report requests to new handlers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20986) Separate the config of block size when we do log splitting and write Hlog
[ https://issues.apache.org/jira/browse/HBASE-20986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571052#comment-16571052 ] Guanghao Zhang commented on HBASE-20986: Pushed to master branch. But there are conflict when apply to branch-2. [~tianjingyun] Please attach a new patch for branch-2. > Separate the config of block size when we do log splitting and write Hlog > - > > Key: HBASE-20986 > URL: https://issues.apache.org/jira/browse/HBASE-20986 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.1.0, 2.0.1, 2.2.0 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20986.master.001.patch, > HBASE-20986.master.002.patch, HBASE-20986.master.003.patch, > HBASE-20986.master.004.patch > > > Since the block size of recovered edits and hlog are the same right now, if > we set a large value to block size, name node may not able to assign enough > space when we do log splitting. But set a large value to hlog block size can > help reduce the number of region server asking for a new block. Thus I think > separate the config of block size is necessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571000#comment-16571000
]
Andrew Purtell commented on HBASE-20704:
Sorry for the delay.
The v4 patch looks like the right thing to do, IMHO.
In testIOExceptionThrownOnClose you have a stray comment here?
{code}
+//Wait a bit for file to be remove from
+try {
+ assertNotNull(streamScanner.next());
+ fail("Expected IOException");
+} catch (IOException ex) {
+ ex.printStackTrace();
+}
{code}
This test isn't actually waiting for anything to happen, right?
{{store.closeAndArchiveCompactedFiles(true);}} completed all actions at that
step. Or otherwise this could flake.
> Sometimes some compacted storefiles are not archived on region close
>
>
> Key: HBASE-20704
> URL: https://issues.apache.org/jira/browse/HBASE-20704
> Project: HBase
> Issue Type: Bug
> Components: Compaction
>Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0
>Reporter: Francis Liu
>Assignee: Francis Liu
>Priority: Critical
> Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch,
> HBASE-20704.003.patch, HBASE-20704.004.draft.patch
>
>
> During region close compacted files which have not yet been archived by the
> discharger are archived as part of the region closing process. It is
> important that these files are wholly archived to insure data consistency. ie
> a storefile containing delete tombstones can be archived while older
> storefiles containing cells that were supposed to be deleted are left
> unarchived thereby undeleting those cells.
> On region close a compacted storefile is skipped from archiving if it has
> read references (ie open scanners). This behavior is correct for when the
> discharger chore runs but on region close consistency is of course more
> important so we should add a special case to ignore any references on the
> storefile and go ahead and archive it.
> Attached patch contains a unit test that reproduces the problem and the
> proposed fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20919) meta region can't be re-onlined when restarting cluster if opening rsgroup
[
https://issues.apache.org/jira/browse/HBASE-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20919:
---
Status: Open (was: Patch Available)
> meta region can't be re-onlined when restarting cluster if opening rsgroup
> --
>
> Key: HBASE-20919
> URL: https://issues.apache.org/jira/browse/HBASE-20919
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master, rsgroup
>Affects Versions: 2.0.1
>Reporter: chenyang
>Assignee: ChenYang
>Priority: Major
> Attachments: HBASE-20919-branch-2.0-01.patch,
> HBASE-20919-branch-2.0-02.patch, HBASE-20919-branch-2.0-02.patch, bug2.png,
> hbase-hbase-master-bjpg-rs4729.yz02.no_02patch.log,
> hbase-hbase-master-bjpg-rs4729.yz02.with_02patch.log,
> hbase-hbase-master-bjpg-rs4730.yz02.log.test
>
>
> if you open rsgroup, hbase-site.xml contains below configuration.
> {code:java}
>
> hbase.coprocessor.master.classes
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint
>
>
> hbase.master.loadbalancer.class/name>
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer
>
> {code}
> And you shut down the whole HBase cluster in the way:
> # first shut down region server one by one
> # shut down master
> Then you restart whole cluster in the way:
> # start master
> # start regionserver
> The hbase:meta region can not be re-online and the rsgroup can not be
> initialized successfully.
> master logs:
> {code:java}
> 2018-07-12 18:27:08,775 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> rsgroup.RSGro
> upInfoManagerImpl$RSGroupStartupWorker: Waiting for catalog tables to come
> online
> 2018-07-12 18:27:08,876 INFO
> [org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-bjpg-rs4730.yz02,16000,1531389637409]
> zookeeper.Met
> aTableLocator: Failed verification of hbase:meta,,1 at
> address=bjpg-rs4732.yz02,60020,1531388712053,
> exception=org.apache.hadoop.hbase.NotServingRegionExcepti
> on: hbase:meta,,1 is not online on bjpg-rs4732.yz02,60020,1531389727928
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3249)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3226)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1414)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1729)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28286)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> The logs show that hbase:meta region is not online and rsgroup keeps retrying
> to initialize.
>
> but why the hbase:meta region is not online?
> The info-level logs and jstack had not enough infomation, so I added some
> debug logs in test-source-code. Then i checked the master`s logs and region
> server`s logs, and found the meta region assign procedure which hold the meta
> region lock not completed and not released the lock forever, so the
> recoverMetaProcedure could not be executed.
>
> Why the first procedure not completed and not released meta region lock?
> In the test logs, i found when assignmentManager assigned the region, it
> need to call the rsgroup balancer which have not been initialized
> completely, so throw NPE. As a result, the procedure not completed and not
> released the lock forever.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:262)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignmentPlans(AssignmentManager.java:1864)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:1809)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$400(AssignmentManager.java:113)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager$2.run(AssignmentManager.java:1693)
> {code}
> !bug2.png!
> As shown in the figure named bug2.png listed in attachments, when we shutdown
> the last region server, the master submit a ServerCrashProcedure. In the
> procedure, it will reassign hbase:meta region, but at that moment, there is
> no
[jira] [Comment Edited] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570992#comment-16570992
]
Andrew Purtell edited comment on HBASE-21011 at 8/7/18 1:05 AM:
Because this changes on the wire messages we need to consider wire
compatibility. I don't think this will work:
{code:java}
message RunCleanerChoreRequest {
+ required CleanerType type = 1;
}
{code}
Instead, CleanerType should be optional with a default value of DEFAULT.
Otherwise newer masters will choke on requests from older admin clients that
will not provide 'type'
was (Author: apurtell):
Because this changes on the wire messages we need to consider wire
compatibility. I don't think this will work:
{code:java}
message RunCleanerChoreRequest {
+ required CleanerType type = 1;
}
{code}
Instead, CleanerType should be optional with a default value of DEFAULT.
Otherwise newer masters will choke on requests from older admin clients that
will not provide 'type' Definitely needed if considering this for commit to
branch-1.
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: HBASE-21011.master.001.patch,
> HBASE-21011.master.002.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570992#comment-16570992
]
Andrew Purtell commented on HBASE-21011:
Because this changes on the wire messages we need to consider wire
compatibility. I don't think this will work:
{code:java}
message RunCleanerChoreRequest {
+ required CleanerType type = 1;
}
{code}
Instead, CleanerType should be optional with a default value of DEFAULT.
Otherwise newer masters will choke on requests from older admin clients that
will not provide 'type' Definitely needed if considering this for commit to
branch-1.
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: HBASE-21011.master.001.patch,
> HBASE-21011.master.002.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21018) RS crashed because AsyncFS was unable to update HDFS data encryption key
[
https://issues.apache.org/jira/browse/HBASE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HBASE-21018:
Attachment: HBASE-21018.master.001.patch
> RS crashed because AsyncFS was unable to update HDFS data encryption key
>
>
> Key: HBASE-21018
> URL: https://issues.apache.org/jira/browse/HBASE-21018
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0, HBase 2.0.0,
> HDFS configuration dfs.encrypt.data.transfer = true
>Reporter: Wei-Chiu Chuang
>Priority: Critical
> Attachments: HBASE-21018.master.001.patch
>
>
> We (+[~uagashe]) found HBase RegionServer doesn't update HDFS data encryption
> key correctly, and in some cases after retry 10 times, it aborts.
> {noformat}
> 2018-08-03 17:37:03,233 WARN
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create
> fan-out dfs output
> /hbase/WALs/rs1.example.com,22101,1533318719239/rs1.example.com%2C22101%2C1533318719239.rs1.example.com%2C22101%2C1533318719239.regiongroup-0.1533343022981
> failed, retry = 1
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=1685436998) doesn't exist. Current key: 1085959374
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.check(FanOutOneBlockAsyncDFSOutputSaslHelper.java:399)
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.channelRead(FanOutOneBlockAsyncDFSOutputSaslHelper.java:470)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
> at
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
> at
>
[jira] [Commented] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570978#comment-16570978 ] huaxiang sun commented on HBASE-20943: -- Hi [~jinghanx], the patch looks good. One nit, can getOnlineRegions/offlineRegion be merged into one method, return PairOfSameType so it can save some cycles? > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Assignee: jinghan xu >Priority: Minor > Attachments: HBASE-20943.patch, Screen Shot 2018-07-25 at 2.51.19 > PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21018) RS crashed because AsyncFS was unable to update HDFS data encryption key
[
https://issues.apache.org/jira/browse/HBASE-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HBASE-21018:
Description:
We (+[~uagashe]) found HBase RegionServer doesn't update HDFS data encryption
key correctly, and in some cases after retry 10 times, it aborts.
{noformat}
2018-08-03 17:37:03,233 WARN
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create
fan-out dfs output
/hbase/WALs/rs1.example.com,22101,1533318719239/rs1.example.com%2C22101%2C1533318719239.rs1.example.com%2C22101%2C1533318719239.regiongroup-0.1533343022981
failed, retry = 1
org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
Can't re-compute encryption key for nonce, since the required block key
(keyID=1685436998) doesn't exist. Current key: 1085959374
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.check(FanOutOneBlockAsyncDFSOutputSaslHelper.java:399)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.channelRead(FanOutOneBlockAsyncDFSOutputSaslHelper.java:470)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
{noformat}
It appears similar to HDFS-10609 and HDFS-11741. There are cases where
dfsclient has to clear its local cache of data encryption key when a DataNode
finds the key expires. Now that asyncfs implements the fan-out block writer,
HBase has to take care of
[jira] [Created] (HBASE-21018) RS crashed because AsyncFS was unable to update HDFS data encryption key
Wei-Chiu Chuang created HBASE-21018:
---
Summary: RS crashed because AsyncFS was unable to update HDFS data
encryption key
Key: HBASE-21018
URL: https://issues.apache.org/jira/browse/HBASE-21018
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 2.0.0
Environment: Hadoop 3.0.0, HBase 2.0.0,
HDFS configuration dfs.encrypt.data.transfer = true
Reporter: Wei-Chiu Chuang
We (+[~uagashe]) found HBase RegionServer doesn't update HDFS data encryption
key correctly, and in some cases after retry 10 times, it aborts.
{noformat}
2018-08-03 17:37:03,233 WARN
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper: create
fan-out dfs output
/hbase/WALs/vb1134.halxg.cloudera.com,22101,1533318719239/vb1134.halxg.cloudera.com%2C22101%2C1533318719239.vb1134.halxg.cloudera.com%2C22101%2C1533318719239.regiongroup-0.1533343022981
failed, retry = 1
org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
Can't re-compute encryption key for nonce, since the required block key
(keyID=1685436998) doesn't exist. Current key: 1085959374
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.check(FanOutOneBlockAsyncDFSOutputSaslHelper.java:399)
at
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper$SaslNegotiateHandler.channelRead(FanOutOneBlockAsyncDFSOutputSaslHelper.java:470)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
at
org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:801)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404)
at
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at
[jira] [Commented] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570976#comment-16570976
]
Hadoop QA commented on HBASE-21011:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} hbase-client: The patch generated 3 new + 417
unchanged - 0 fixed = 420 total (was 417) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 15 new + 11
unchanged - 0 fixed = 26 total (was 11) {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
9s{color} | {color:red} The patch generated 6 new + 389 unchanged - 0 fixed =
395 total (was 389) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 4s{color} | {color:orange} The patch generated 1 new + 728 unchanged - 1
fixed = 729 total (was 729) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 10s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
21s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}125m
50s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570973#comment-16570973
]
Hadoop QA commented on HBASE-21005:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
13s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 1s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}192m
53s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}268m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21005 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12934534/HBASE-21005.001.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile mvnsite |
| uname | Linux ece6fc01900f 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / bd30ca62ef |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13941/testReport/ |
| Max. process+thread count | 5053 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13941/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
>
[jira] [Updated] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinghan xu updated HBASE-20943: --- Attachment: HBASE-20943.patch > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Assignee: jinghan xu >Priority: Minor > Attachments: HBASE-20943.patch, Screen Shot 2018-07-25 at 2.51.19 > PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[
https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570884#comment-16570884
]
Hudson commented on HBASE-20997:
Results for branch branch-1.3
[build #415 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/415/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/415//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/415//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/415//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> rebuildUserRegions() does not build ReplicaMapping during master switchover
> ---
>
> Key: HBASE-20997
> URL: https://issues.apache.org/jira/browse/HBASE-20997
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7
>
> Attachments: HBASE-20997-branch-1-v1.patch,
> HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch,
> HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch
>
>
> During master switchover, rebuildUserRegions() does not rebuild master
> in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an
> inconsistent state. In read replica case, it causes replica parent region
> stay online without being unassigned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20813) Remove RPC quotas when the associated table/Namespace is dropped off
[
https://issues.apache.org/jira/browse/HBASE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sakthi updated HBASE-20813:
---
Attachment: hbase-20813.master.002.patch
> Remove RPC quotas when the associated table/Namespace is dropped off
>
>
> Key: HBASE-20813
> URL: https://issues.apache.org/jira/browse/HBASE-20813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20813.master.001.patch,
> hbase-20813.master.002.patch
>
>
> In short, the below scenario shouldn't be the case.
> {noformat}
> hbase(main):023:0> create 't2','cf1'
> Created table t2
> Took 0.7405 seconds
> => Hbase::Table - t2
> hbase(main):024:0>
> hbase(main):025:0*
> hbase(main):026:0* set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0082 seconds
> hbase(main):027:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0291 seconds
> hbase(main):028:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0037 seconds
> hbase(main):029:0> disable 't2'
> Took 0.4328 seconds
> hbase(main):030:0> drop 't2'
> Took 0.2285 seconds
> hbase(main):031:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0230 seconds
> hbase(main):032:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0038 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21017) Revisit the expected states for open/close
Duo Zhang created HBASE-21017: - Summary: Revisit the expected states for open/close Key: HBASE-21017 URL: https://issues.apache.org/jira/browse/HBASE-21017 Project: HBase Issue Type: Sub-task Components: amv2 Reporter: Duo Zhang Fix For: 3.0.0, 2.2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20990) One operation in procedure batch throws an exception will cause all RegionTransitionProcedures receive the same exception
[
https://issues.apache.org/jira/browse/HBASE-20990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570866#comment-16570866
]
Duo Zhang commented on HBASE-20990:
---
If the RS restarts then it is the turn for SCP. And in fact, if the RS is dead,
in the sync call you can do nothing either, you will get a connection refuse or
some other exceptions, and it is not clear that whether this is a RS crash or a
network error. SCP is the only way to change the target server.
And if the rpc call is successfully returned then we can make sure that the
procedure is executing at RS side, just in a background thread pool. It is the
duty for the RS to make sure that you need to report back to master the result,
unless you are dead.
> One operation in procedure batch throws an exception will cause all
> RegionTransitionProcedures receive the same exception
> -
>
> Key: HBASE-20990
> URL: https://issues.apache.org/jira/browse/HBASE-20990
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Affects Versions: 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
>
> In AMv2, we batch open/close region operations and call RS with
> executeProcedures API. But, in this API, if one of the region's operations
> throws an exception, all the operations in the batch will receive the same
> exception. Actually, some of the operations in the batch is executing
> normally in the RS.
> I think we should try catch exceptions respectively, and call
> remoteCallFailed or remoteCallCompleted in RegionTransitionProcedure
> respectively.
> Otherwise, there will be some very strange behave. Such as this one:
> {code}
> 2018-07-18 02:56:18,506 WARN [RSProcedureDispatcher-pool3-t1]
> assignment.RegionTransitionProcedure(226): Remote call failed
> e010125048016.bja,60020,1531848989401; pid=8362, ppid=8272, state=RUNNABLE:R
> EGION_TRANSITION_DISPATCH; AssignProcedure
> table=IntegrationTestBigLinkedList, region=0beb8ea4e2f239fc082be7cefede1427,
> target=e010125048016.bja,60020,1531848989401; rit=OPENING,
> location=e010125048016
> .bja,60020,1531848989401; exception=NotServingRegionException
> {code}
> The AssignProcedure failed with a NotServingRegionException, what??? It is
> very strange, actually, the AssignProcedure successes on the RS, another
> CloseRegion operation failed in the operation batch was causing the exception.
> To correct this, we need to modify the response of executeProcedures API,
> which is the ExecuteProceduresResponse proto, to return infos(status,
> exceptions) per operation.
> This issue alone won't cause much trouble, so not so hurry to change the
> behave here, but indeed we need to consider this one when we want do some
> reconstruct to AMv2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21016) Re-enable TestAssignmentManager and TestUnexpectedStateException after HBASE-20881
Duo Zhang created HBASE-21016: - Summary: Re-enable TestAssignmentManager and TestUnexpectedStateException after HBASE-20881 Key: HBASE-21016 URL: https://issues.apache.org/jira/browse/HBASE-21016 Project: HBase Issue Type: Sub-task Components: amv2, test Reporter: Duo Zhang Fix For: 3.0.0, 2.2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[ https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20881: -- Attachment: HBASE-20881-v4.patch > Introduce a region transition procedure to handle all the state transition > for a region > --- > > Key: HBASE-20881 > URL: https://issues.apache.org/jira/browse/HBASE-20881 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20881-v1.patch, HBASE-20881-v2.patch, > HBASE-20881-v3.patch, HBASE-20881-v4.patch, HBASE-20881.patch > > > Now have an AssignProcedure, an UnssignProcedure, and also a > MoveRegionProcedure which schedules an AssignProcedure and an > UnssignProcedure to move a region. This makes the logic a bit complicated, as > MRP is not a RIT, so when SCP can not interrupt it directly... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20958) Port WAL recovery code to use the new WAL API
[ https://issues.apache.org/jira/browse/HBASE-20958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal reassigned HBASE-20958: - Assignee: Ankit Singhal > Port WAL recovery code to use the new WAL API > - > > Key: HBASE-20958 > URL: https://issues.apache.org/jira/browse/HBASE-20958 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Ankit Singhal >Priority: Major > > The recovery code needs to be reworked into the new WAL API > Might be tricky because we have the WALSplitter now which is not something > that is "constant". > One thought was that we could support per-Region-WALs instead of per-RS-WALs > with the Ratis-LogService WALs. We would need to figure out how to wrangle > this inside of an implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20957) Switch replication over to use the new WAL API
[ https://issues.apache.org/jira/browse/HBASE-20957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570850#comment-16570850 ] Ankit Singhal commented on HBASE-20957: --- Started working on it. At a high level, it looks, * Need to abstract WAL name under WalInfo instead of Paths * Abstract the WalEntryStream for FileSystem and Streaming system. * Build WalStorage APIs to abstract operation on Wal. * Provide the implementation of all above through corresponding WalProvider. > Switch replication over to use the new WAL API > -- > > Key: HBASE-20957 > URL: https://issues.apache.org/jira/browse/HBASE-20957 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Ankit Singhal >Priority: Major > > Replication needs to use the new WAL APIs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20957) Switch replication over to use the new WAL API
[ https://issues.apache.org/jira/browse/HBASE-20957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Singhal reassigned HBASE-20957: - Assignee: Ankit Singhal > Switch replication over to use the new WAL API > -- > > Key: HBASE-20957 > URL: https://issues.apache.org/jira/browse/HBASE-20957 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Ankit Singhal >Priority: Major > > Replication needs to use the new WAL APIs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-21011:
-
Fix Version/s: (was: 3.0.0)
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: HBASE-21011.master.001.patch,
> HBASE-21011.master.002.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-21011:
-
Attachment: HBASE-21011.master.002.patch
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21011.master.001.patch,
> HBASE-21011.master.002.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21007) Memory leak in HBase rest server
[ https://issues.apache.org/jira/browse/HBASE-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570802#comment-16570802 ] Andrew Purtell commented on HBASE-21007: Please fix the checkstyle warnings, then we can commit this. > Memory leak in HBase rest server > > > Key: HBASE-21007 > URL: https://issues.apache.org/jira/browse/HBASE-21007 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 1.4.0, 1.4.6 >Reporter: Bosko Devetak >Assignee: Bosko Devetak >Priority: Critical > Attachments: HBASE-21007.001.patch > > > When using the URIs like this: > > /sometable/*?limit=$limit=$startrow=$endrow > > where *$limit* is smaller than the range between *$startrow* and *$endrow*, > the rest server will start leaking memory. > > > The bug is in the *TableScanResource.java* class. Basically, the > ResultScanner is not being closed in next() method when the limit has been > reached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20753) reference guide should direct security related issues to priv...@hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570779#comment-16570779
]
Hadoop QA commented on HBASE-20753:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
7s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
5s{color} | {color:blue} patch has no errors when building the reference guide.
See footer for rendered docs, which you should manually inspect. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20753 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12933505/HBASE-20753.master.001.patch
|
| Optional Tests | asflicense refguide |
| uname | Linux 273b0d5b3f4c 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / bd30ca62ef |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13942/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13942/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 83 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13942/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> reference guide should direct security related issues to
> priv...@hbase.apache.org
> -
>
> Key: HBASE-20753
> URL: https://issues.apache.org/jira/browse/HBASE-20753
> Project: HBase
> Issue Type: Bug
> Components: documentation, security
>Reporter: Sean Busbey
>Priority: Critical
> Labels: beginner
> Attachments: HBASE-20753.master.001.patch
>
>
> the reference guide currently directs folks to send security issues to
> priv...@apache.org:
> {quote}
> To protect existing HBase installations from new vulnerabilities, please do
> not use JIRA to report security-related bugs. Instead, send your report to
> the mailing list priv...@apache.org, which allows anyone to send messages,
> but restricts who can read them. Someone on that list will contact you to
> follow up on your report.
> {quote}
> This address does not exist. It should tell folks to send the email to
> priv...@hbase.apache.org.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-20943: -- Assignee: jinghan xu > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Assignee: jinghan xu >Priority: Minor > Attachments: Screen Shot 2018-07-25 at 2.51.19 PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20753) reference guide should direct security related issues to priv...@hbase.apache.org
[
https://issues.apache.org/jira/browse/HBASE-20753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HBASE-20753:
--
Status: Patch Available (was: Open)
> reference guide should direct security related issues to
> priv...@hbase.apache.org
> -
>
> Key: HBASE-20753
> URL: https://issues.apache.org/jira/browse/HBASE-20753
> Project: HBase
> Issue Type: Bug
> Components: documentation, security
>Reporter: Sean Busbey
>Priority: Critical
> Labels: beginner
> Attachments: HBASE-20753.master.001.patch
>
>
> the reference guide currently directs folks to send security issues to
> priv...@apache.org:
> {quote}
> To protect existing HBase installations from new vulnerabilities, please do
> not use JIRA to report security-related bugs. Instead, send your report to
> the mailing list priv...@apache.org, which allows anyone to send messages,
> but restricts who can read them. Someone on that list will contact you to
> follow up on your report.
> {quote}
> This address does not exist. It should tell folks to send the email to
> priv...@hbase.apache.org.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[
https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570737#comment-16570737
]
Hudson commented on HBASE-20997:
Results for branch branch-1.2
[build #420 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/420/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/420//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/420//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/420//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> rebuildUserRegions() does not build ReplicaMapping during master switchover
> ---
>
> Key: HBASE-20997
> URL: https://issues.apache.org/jira/browse/HBASE-20997
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7
>
> Attachments: HBASE-20997-branch-1-v1.patch,
> HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch,
> HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch
>
>
> During master switchover, rebuildUserRegions() does not rebuild master
> in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an
> inconsistent state. In read replica case, it causes replica parent region
> stay online without being unassigned.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20813) Remove RPC quotas when the associated table/Namespace is dropped off
[
https://issues.apache.org/jira/browse/HBASE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570735#comment-16570735
]
Sakthi commented on HBASE-20813:
Got it [~elserj]. I'll rename the file and add RPC quota's handling inside the
same file as well and submit a new patch.
> Remove RPC quotas when the associated table/Namespace is dropped off
>
>
> Key: HBASE-20813
> URL: https://issues.apache.org/jira/browse/HBASE-20813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20813.master.001.patch
>
>
> In short, the below scenario shouldn't be the case.
> {noformat}
> hbase(main):023:0> create 't2','cf1'
> Created table t2
> Took 0.7405 seconds
> => Hbase::Table - t2
> hbase(main):024:0>
> hbase(main):025:0*
> hbase(main):026:0* set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0082 seconds
> hbase(main):027:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0291 seconds
> hbase(main):028:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0037 seconds
> hbase(main):029:0> disable 't2'
> Took 0.4328 seconds
> hbase(main):030:0> drop 't2'
> Took 0.2285 seconds
> hbase(main):031:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0230 seconds
> hbase(main):032:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0038 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-21005:
---
Status: Patch Available (was: In Progress)
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
> Attachments: HBASE-21005.001.patch
>
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-21005:
---
Attachment: HBASE-21005.001.patch
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
> Attachments: HBASE-21005.001.patch
>
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570676#comment-16570676
]
Josh Elser commented on HBASE-21005:
Everything looks good using the SNAPSHOT, so I went ahead and released my
forked repo to sonatype. Waiting on this to propagate to central, and then will
flip this sucka over to patch available.
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570665#comment-16570665 ] Tianying Chang commented on HBASE-20943: [~yuzhih...@gmail.com] it seems I cannot assign this Jira to [~jinghanx], any permission needed? > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Priority: Minor > Attachments: Screen Shot 2018-07-25 at 2.51.19 PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570663#comment-16570663 ] jinghan xu commented on HBASE-20943: hey [~yuzhih...@gmail.com], can you help adding me into the project? > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Priority: Minor > Attachments: Screen Shot 2018-07-25 at 2.51.19 PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[ https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570645#comment-16570645 ] Hudson commented on HBASE-20997: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #448 (See [https://builds.apache.org/job/HBase-1.3-IT/448/]) HBASE-20997 rebuildUserRegions() does not build ReplicaMapping during (hsun: rev 7b8ac0471569a59f5dad463915347eb3ce69f75c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailoverBalancerPersistence.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java > rebuildUserRegions() does not build ReplicaMapping during master switchover > --- > > Key: HBASE-20997 > URL: https://issues.apache.org/jira/browse/HBASE-20997 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7 > > Attachments: HBASE-20997-branch-1-v1.patch, > HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch, > HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch > > > During master switchover, rebuildUserRegions() does not rebuild master > in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an > inconsistent state. In read replica case, it causes replica parent region > stay online without being unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[ https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570636#comment-16570636 ] Hudson commented on HBASE-20997: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1144 (See [https://builds.apache.org/job/HBase-1.2-IT/1144/]) HBASE-20997 rebuildUserRegions() does not build ReplicaMapping during (hsun: rev ba0cc42bd63309a9e67c1c71e8895a9fbd69b2e4) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailoverBalancerPersistence.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java > rebuildUserRegions() does not build ReplicaMapping during master switchover > --- > > Key: HBASE-20997 > URL: https://issues.apache.org/jira/browse/HBASE-20997 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7 > > Attachments: HBASE-20997-branch-1-v1.patch, > HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch, > HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch > > > During master switchover, rebuildUserRegions() does not rebuild master > in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an > inconsistent state. In read replica case, it causes replica parent region > stay online without being unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region
[ https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun resolved HBASE-20651. -- Resolution: Fixed > Master, prevents hbck or shell command to reassign the split parent region > -- > > Key: HBASE-20651 > URL: https://issues.apache.org/jira/browse/HBASE-20651 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6 > > Attachments: HBASE-20651-branch-1-v001.patch, > HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch > > > We are seeing that hbck brings back split parent region and this causes > region inconsistency. More details will be filled as reproduce is still > ongoing. Might need to do something at hbck or master to prevent this from > happening. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region
[ https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570630#comment-16570630 ] huaxiang sun commented on HBASE-20651: -- Does not seem a regression. The issue we were looking at is HBASE-20997, resolving it. > Master, prevents hbck or shell command to reassign the split parent region > -- > > Key: HBASE-20651 > URL: https://issues.apache.org/jira/browse/HBASE-20651 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6 > > Attachments: HBASE-20651-branch-1-v001.patch, > HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch > > > We are seeing that hbck brings back split parent region and this causes > region inconsistency. More details will be filled as reproduce is still > ongoing. Might need to do something at hbck or master to prevent this from > happening. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[ https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570625#comment-16570625 ] huaxiang sun commented on HBASE-20997: -- Thanks [~yuzhih...@gmail.com]. I pushed the patch to 1.2, 1.3, 1.4 and branch-1. Thanks [~pankaj2461] for the review. > rebuildUserRegions() does not build ReplicaMapping during master switchover > --- > > Key: HBASE-20997 > URL: https://issues.apache.org/jira/browse/HBASE-20997 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7 > > Attachments: HBASE-20997-branch-1-v1.patch, > HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch, > HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch > > > During master switchover, rebuildUserRegions() does not rebuild master > in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an > inconsistent state. In read replica case, it causes replica parent region > stay online without being unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[ https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-20997: - Resolution: Fixed Fix Version/s: 1.4.7 1.3.3 1.2.7 1.5.0 Status: Resolved (was: Patch Available) Pushed the diff to fixed branches. > rebuildUserRegions() does not build ReplicaMapping during master switchover > --- > > Key: HBASE-20997 > URL: https://issues.apache.org/jira/browse/HBASE-20997 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.7 > > Attachments: HBASE-20997-branch-1-v1.patch, > HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch, > HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch > > > During master switchover, rebuildUserRegions() does not rebuild master > in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an > inconsistent state. In read replica case, it causes replica parent region > stay online without being unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570601#comment-16570601
]
Tak Lon (Stephen) Wu commented on HBASE-21011:
--
thanks [~busbey], before I see any update on the integrating PR with JIRA, I'm
going to use review board and attached a patch.
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21011.master.001.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu updated HBASE-21011:
-
Attachment: HBASE-21011.master.001.patch
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-21011.master.001.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[ https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570582#comment-16570582 ] Ted Yu commented on HBASE-20997: Focusing this JIRA on branch-1.x is fine. > rebuildUserRegions() does not build ReplicaMapping during master switchover > --- > > Key: HBASE-20997 > URL: https://issues.apache.org/jira/browse/HBASE-20997 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Attachments: HBASE-20997-branch-1-v1.patch, > HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch, > HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch > > > During master switchover, rebuildUserRegions() does not rebuild master > in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an > inconsistent state. In read replica case, it causes replica parent region > stay online without being unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20997) rebuildUserRegions() does not build ReplicaMapping during master switchover
[ https://issues.apache.org/jira/browse/HBASE-20997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570577#comment-16570577 ] huaxiang sun commented on HBASE-20997: -- Hi [~yuzhih...@gmail.com], for branch-2, getRegionAssignments() is using regionMap, which is built within loadMeta(). This is different from branch-1, it builds a specialMap for that. Regarding with regression, if you think the test is needed to make sure that regionMap is consistent across master failover for branch-2, I can create a new jira to track it. Thanks. > rebuildUserRegions() does not build ReplicaMapping during master switchover > --- > > Key: HBASE-20997 > URL: https://issues.apache.org/jira/browse/HBASE-20997 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.2.6, 1.3.2, 1.5.0, 1.4.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Major > Attachments: HBASE-20997-branch-1-v1.patch, > HBASE-20997-branch-1-v2.patch, HBASE-20997-branch-1-v4.patch, > HBASE-20997-branch-1-v5.patch, HBASE-20997-branch-1-v6.patch > > > During master switchover, rebuildUserRegions() does not rebuild master > in-memory defaultReplicaToOtherReplicas map. This puts the cluster in an > inconsistent state. In read replica case, it causes replica parent region > stay online without being unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20943) Add offline/online region count into metrics
[ https://issues.apache.org/jira/browse/HBASE-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570561#comment-16570561 ] Tianying Chang commented on HBASE-20943: [~yuzhih...@gmail.com] Thanks. My teammate Jinghan Xu will upload the patch. > Add offline/online region count into metrics > > > Key: HBASE-20943 > URL: https://issues.apache.org/jira/browse/HBASE-20943 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0, 1.2.6.1 >Reporter: Tianying Chang >Priority: Minor > Attachments: Screen Shot 2018-07-25 at 2.51.19 PM.png > > > We intensively use metrics to monitor the health of our HBase production > cluster. We have seen some regions of a table stuck and cannot be brought > online due to AWS issue which cause some log file corrupted. It will be good > if we can catch this early. Although WebUI has this information, it is not > useful for automated monitoring. By adding this metric, we can easily monitor > them with our monitoring system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[
https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570530#comment-16570530
]
Hadoop QA commented on HBASE-20881:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} hbase-client: The patch generated 0 new + 2
unchanged - 84 fixed = 2 total (was 86) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} hbase-procedure: The patch generated 0 new + 20
unchanged - 1 fixed = 20 total (was 21) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} hbase-server: The patch generated 0 new + 253
unchanged - 49 fixed = 253 total (was 302) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 33s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
31s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green}
[jira] [Commented] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570507#comment-16570507
]
Hadoop QA commented on HBASE-19008:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} hbase-client: The patch generated 94 new + 353
unchanged - 68 fixed = 447 total (was 421) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
21s{color} | {color:red} hbase-server: The patch generated 11 new + 404
unchanged - 2 fixed = 415 total (was 406) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hbase-spark: The patch generated 22 new + 0 unchanged
- 0 fixed = 22 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
4s{color} | {color:red} hbase-client generated 2 new + 0 unchanged - 0 fixed =
2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
11s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 59s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
21s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
6s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-client |
| |
org.apache.hadoop.hbase.filter.SingleColumnValueExcludeFilter.equals(Object)
compares class names rather than
[jira] [Commented] (HBASE-20749) Upgrade our use of checkstyle to 8.6+
[
https://issues.apache.org/jira/browse/HBASE-20749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570495#comment-16570495
]
Hudson commented on HBASE-20749:
Results for branch HBASE-20749
[build #15 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/15/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/15//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/15//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20749/15//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upgrade our use of checkstyle to 8.6+
> -
>
> Key: HBASE-20749
> URL: https://issues.apache.org/jira/browse/HBASE-20749
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Minor
> Attachments: HBASE-20749.master.001.patch,
> HBASE-20749.master.002.patch, HBASE-20749.master.003.patch
>
>
> We should upgrade our checkstyle version to 8.6 or later so we can use the
> "match violation message to this regex" feature for suppression. That will
> allow us to make sure we don't regress on HTrace v3 vs v4 APIs (came up in
> HBASE-20332).
> We're currently blocked on upgrading to 8.3+ by [checkstyle
> #5279|https://github.com/checkstyle/checkstyle/issues/5279], a regression
> that flags our use of both the "separate import groups" and "put static
> imports over here" configs as an error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570455#comment-16570455
]
Josh Elser commented on HBASE-20952:
{quote}Would suggest starting review of existing implementations. Just reading
[https://bookkeeper.apache.org/docs/latest/api/distributedlog-api/] is
interesting ("Single writer!") or namespaces – is a namespace a "region" in our
work – in here
[https://bookkeeper.apache.org/distributedlog/docs/latest/user_guide/api/core]
{quote}
FYI, I know that [~chrajeshbab...@gmail.com] is looking at
BookKeeper/DistributedLog specifically as a part of RATIS-272. Plan is to
converge that work with this one here.
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Josh Elser
>Priority: Major
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570435#comment-16570435
]
Sean Busbey commented on HBASE-21011:
-
It looks like there are two issues.
1) There's a gap in what the admin job does. It only tries to launch jobs when
there's an attachment. we'll need to file a request in the yetus project to
either have that job also look for PRs, or get a different admin job that
specifically looks for PRs.
2) manually triggering a precommit build against this issue fails when trying
to get the PR, but even with debug it doesn't say what. we'll need to dig in by
running test-patch locally to figure out what's going on.
I don't think you need to wait on either of these; whoever wants to address
them can make a placeholder issue. For now, you'll need to attach the patch
here to get a qabot run.
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570429#comment-16570429
]
Josh Elser commented on HBASE-21005:
If anyone wants to validate the staged variant:
https://static.penguinsinabox.com/files/target/site/
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570424#comment-16570424
]
Sean Busbey commented on HBASE-21011:
-
link to PR in comment text: https://github.com/apache/hbase/pull/89
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20813) Remove RPC quotas when the associated table/Namespace is dropped off
[
https://issues.apache.org/jira/browse/HBASE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570422#comment-16570422
]
Josh Elser commented on HBASE-20813:
{quote}I thought this suggestion meant, to keep MasterSpaceQuotaObserver for
backwards-compat issues?
{quote}
Well, we can keep the old one around, but we definitely shouldn't be
duplicating code as this patch does.
Anyways, looking at it again, we don't have users set this manually, instead
HBase configures it automatically (unless the users tells us not to). As such,
it's probably not a big deal to rename it instead of keeping the old impl
around (there's no reason I can think of that someone would _only_ want to drop
the table for space quotas but not RPC quotas).
> Remove RPC quotas when the associated table/Namespace is dropped off
>
>
> Key: HBASE-20813
> URL: https://issues.apache.org/jira/browse/HBASE-20813
> Project: HBase
> Issue Type: Sub-task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Attachments: hbase-20813.master.001.patch
>
>
> In short, the below scenario shouldn't be the case.
> {noformat}
> hbase(main):023:0> create 't2','cf1'
> Created table t2
> Took 0.7405 seconds
> => Hbase::Table - t2
> hbase(main):024:0>
> hbase(main):025:0*
> hbase(main):026:0* set_quota TYPE => THROTTLE, TABLE => 't2', LIMIT =>
> '10M/sec'
> Took 0.0082 seconds
> hbase(main):027:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0291 seconds
> hbase(main):028:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0037 seconds
> hbase(main):029:0> disable 't2'
> Took 0.4328 seconds
> hbase(main):030:0> drop 't2'
> Took 0.2285 seconds
> hbase(main):031:0> list_quotas
> OWNER QUOTAS
> TABLE => t2 TYPE => THROTTLE, THROTTLE_TYPE => REQUEST_SIZE, LIMIT =>
> 10M/sec, SCOPE => MACHINE
> 1 row(s)
> Took 0.0230 seconds
> hbase(main):032:0> scan 'hbase:quota'
> ROW COLUMN+CELL
> t.t2 column=q:s, timestamp=1530165010888,
> value=PBUF\x12\x0B\x12\x09\x08\x04\x10\x80\x80\x80\x05 \x02
> 1 row(s)
> Took 0.0038 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570395#comment-16570395
]
Josh Elser commented on HBASE-21005:
{quote}It's because [https://github.com/apache/hbase/blob/master/pom.xml#L3806]
doesn't get proper property substitution, I think.
{quote}
Ya, that's the working-assumption, [~mdrob]. Getting rid of this JAR from our
source repo is a win as well, IMO. While we could probably "fix" this with some
re-jiggering of the POM, I think we can kill two birds with one stone just
getting this JAR published normally.
Thanks [~mattyb149]!
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21015) The state of the split parent region is wrong after cloning a snapshot
Toshihiro Suzuki created HBASE-21015:
Summary: The state of the split parent region is wrong after
cloning a snapshot
Key: HBASE-21015
URL: https://issues.apache.org/jira/browse/HBASE-21015
Project: HBase
Issue Type: Bug
Reporter: Toshihiro Suzuki
Assignee: Toshihiro Suzuki
Attachments: Screen Shot 2018-08-07 at 00.40.40.png
Steps to reproduce are as follows:
1. Create a table
{code}
create "test", "cf"
{code}
2. Put some date into the table
{code}
(0...2000).each{|i| put "test", "row#{i}", "cf:col", "val"}
{code}
3. Split the table
{code}
split "test"
{code}
4. Take a snapshot of the table before CatalogJanitor cleans up the split
parent region
{code}
snapshot "test", "snap"
{code}
5. Clone the snapshot
{code}
clone_snapshot "snap", "cloned_table"
{code}
After cloning the snapshot, the state of the split parent region is OFFLINE,
but it should be SPLIT.
!Screen Shot 2018-08-07 at 00.40.40.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570367#comment-16570367
]
Matt Burgess commented on HBASE-21005:
--
If you have a NiFi GitHub fork/clone, you can just do a mvn clean install from
the nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors directory. If you
have a clean maven repo (or at least that glassfish javax.el part you can see
in the maven output), then it currently creates a directory called
${project.basedir}. Doing a maven install without that artifact in the local
repo, you shouldn't see that directory created. I can also do a test on a full
NiFi build (as the directory gets created at the top level currently) unless
you want to :)
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570366#comment-16570366
]
Mike Drob commented on HBASE-21005:
---
It's because https://github.com/apache/hbase/blob/master/pom.xml#L3806 doesn't
get proper property substitution, I think.
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570365#comment-16570365
]
Josh Elser commented on HBASE-21005:
I'm taking Misty's patches over maven-fluido-skin-1.4, forking them to my
Github, and then deploying it to Sonatype's OSSRH.
I've deployed a SNAPSHOT now and am testing building out the website. If this
seems to work, I'll then publish a release and throw a patch up here.
[~mattyb149] I know you gave me some loose steps in private chat: do you have
specific things I could run so that I can test and make sure I've nipped this
in the bud fully?
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HBASE-21005) Maven site configuration causes downstream projects to get a directory named ${project.basedir}
[
https://issues.apache.org/jira/browse/HBASE-21005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-21005 started by Josh Elser.
--
> Maven site configuration causes downstream projects to get a directory named
> ${project.basedir}
> ---
>
> Key: HBASE-21005
> URL: https://issues.apache.org/jira/browse/HBASE-21005
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Matt Burgess
>Assignee: Josh Elser
>Priority: Minor
>
> Matt told me about this interesting issue they see down in Apache Nifi's build
> NiFi depends on HBase for some code that they provide to their users. As a
> part of the build process of NiFi, they are seeing a directory named
> {{$\{project.basedir}}} get created the first time they build with an empty
> Maven repo. Matt reports that after a javax.el artifact is cached, Maven will
> stop creating the directory; however, if you wipe that artifact from the
> Maven repo, the next build will end up re-creating it.
> I believe I've seen this with Phoenix, too, but never investigated why it was
> actually happening.
> My hunch is that it's related to the local maven repo that we create to
> "patch" in our custom maven-fluido-skin jar (HBASE-14785). I'm not sure if we
> can "work" around this by pushing the custom local repo into a profile and
> only activating that for the mvn-site. Another solution would be to publish
> the maven-fluido-jar to central with custom coordinates.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305
[
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570311#comment-16570311
]
Sean Busbey commented on HBASE-20257:
-
thanks for the reminder Ted. In my queue for this week.
> hbase-spark should not depend on com.google.code.findbugs.jsr305
>
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
> Issue Type: Task
> Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch,
> HBASE-20257.v03.patch, HBASE-20257.v04.patch, HBASE-20257.v05.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
>
> com.google.code.findbugs
> jsr305
> {code}
> Dependency on jsr305 should be dropped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570292#comment-16570292
]
Sean Busbey commented on HBASE-21011:
-
I don't see an attempted run in the [HBase Precommit
job|https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-HBASE-Build/]
for this jira key. Unfortunately, the [Precommit Admin
job|https://builds.apache.org/view/PreCommit%20Builds/job/PreCommit-Admin/]
that looks for updated issues in Patch Available and schedules job only keeps
logs for 2 days, so the run for this check is just out of reach at this point.
lemme try getting it to fire again.
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[ https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570243#comment-16570243 ] Duo Zhang commented on HBASE-20881: --- Review board link: https://reviews.apache.org/r/68233/ > Introduce a region transition procedure to handle all the state transition > for a region > --- > > Key: HBASE-20881 > URL: https://issues.apache.org/jira/browse/HBASE-20881 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20881-v1.patch, HBASE-20881-v2.patch, > HBASE-20881-v3.patch, HBASE-20881.patch > > > Now have an AssignProcedure, an UnssignProcedure, and also a > MoveRegionProcedure which schedules an AssignProcedure and an > UnssignProcedure to move a region. This makes the logic a bit complicated, as > MRP is not a RIT, so when SCP can not interrupt it directly... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[ https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570244#comment-16570244 ] Duo Zhang commented on HBASE-20881: --- [~stack] PTAL. Thanks. > Introduce a region transition procedure to handle all the state transition > for a region > --- > > Key: HBASE-20881 > URL: https://issues.apache.org/jira/browse/HBASE-20881 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20881-v1.patch, HBASE-20881-v2.patch, > HBASE-20881-v3.patch, HBASE-20881.patch > > > Now have an AssignProcedure, an UnssignProcedure, and also a > MoveRegionProcedure which schedules an AssignProcedure and an > UnssignProcedure to move a region. This makes the logic a bit complicated, as > MRP is not a RIT, so when SCP can not interrupt it directly... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[ https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20881: -- Attachment: HBASE-20881-v3.patch > Introduce a region transition procedure to handle all the state transition > for a region > --- > > Key: HBASE-20881 > URL: https://issues.apache.org/jira/browse/HBASE-20881 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20881-v1.patch, HBASE-20881-v2.patch, > HBASE-20881-v3.patch, HBASE-20881.patch > > > Now have an AssignProcedure, an UnssignProcedure, and also a > MoveRegionProcedure which schedules an AssignProcedure and an > UnssignProcedure to move a region. This makes the logic a bit complicated, as > MRP is not a RIT, so when SCP can not interrupt it directly... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570216#comment-16570216
]
liubangchen commented on HBASE-19008:
-
Fix issues for FindBugs and CheckStyle.
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-2.patch,
> HBASE-19008-3.patch, HBASE-19008-4.patch, HBASE-19008-5.patch,
> HBASE-19008-6.patch, HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen updated HBASE-19008:
Attachment: HBASE-19008-6.patch
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-2.patch,
> HBASE-19008-3.patch, HBASE-19008-4.patch, HBASE-19008-5.patch,
> HBASE-19008-6.patch, HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[
https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570152#comment-16570152
]
Hadoop QA commented on HBASE-20881:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 18 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} hbase-client: The patch generated 0 new + 2
unchanged - 84 fixed = 2 total (was 86) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} hbase-procedure: The patch generated 0 new + 20
unchanged - 1 fixed = 20 total (was 21) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 6 new + 253
unchanged - 49 fixed = 259 total (was 302) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
30s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
2s{color} | {color:green} hbase-client in
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[
https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570122#comment-16570122
]
Kuan-Po Tseng commented on HBASE-18201:
---
[~reidchan] Thanks for your review.
{quote} Is {{meta.getDataBlockEncoding()}} equals {{this.encoding}} (it looks
so to me). If so, {{this.encoding = encoding}} can be removed IMO.
{quote}
Yes. meta.getDataBlockEncoding == encodingCtx.getDataBlockEncoding() ==
encoding.
But
{code:java}
meta.getDataBlockEncoding().equals(DataBlockEncoding.ROW_INDEX_V1){code}
is needed.
We can change the code to
{code:java}
encoding.equals(DataBlockEncoding.ROW_INDEX_V1)
{code}
It seems more clear.
> add UT and docs for DataBlockEncodingTool
> -
>
> Key: HBASE-18201
> URL: https://issues.apache.org/jira/browse/HBASE-18201
> Project: HBase
> Issue Type: Sub-task
> Components: tooling
>Reporter: Chia-Ping Tsai
>Assignee: Kuan-Po Tseng
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-18201.master.001.patch,
> HBASE-18201.master.002.patch, HBASE-18201.master.002.patch,
> HBASE-18201.master.003.patch, HBASE-18201.master.004.patch,
> HBASE-18201.master.005.patch, HBASE-18201.master.005.patch,
> HBASE-18201.master.005.patch
>
>
> There is no example, documents, or tests for DataBlockEncodingTool. We should
> have it friendly if any use case exists. Otherwise, we should just get rid of
> it because DataBlockEncodingTool presumes that the implementation of cell
> returned from DataBlockEncoder is KeyValue. The presume may obstruct the
> cleanup of KeyValue references in the code base of read/write path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570106#comment-16570106
]
Hadoop QA commented on HBASE-19008:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
1s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 10 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
27s{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 52s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} hbase-client: The patch generated 23 new + 414
unchanged - 7 fixed = 437 total (was 421) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
20s{color} | {color:red} hbase-server: The patch generated 20 new + 418
unchanged - 0 fixed = 438 total (was 418) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hbase-spark: The patch generated 1 new + 0 unchanged -
0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 51s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red} hbase-client generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
2s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}120m
20s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
44s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
59s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Created] (HBASE-21014) Improve Stochastic Balancer to write HDFS favoured node hints for region primary blocks to avoid destroying data locality if needing to use HDFS Balancer
Hari Sekhon created HBASE-21014:
---
Summary: Improve Stochastic Balancer to write HDFS favoured node
hints for region primary blocks to avoid destroying data locality if needing to
use HDFS Balancer
Key: HBASE-21014
URL: https://issues.apache.org/jira/browse/HBASE-21014
Project: HBase
Issue Type: Improvement
Components: Balancer
Affects Versions: 1.1.2
Reporter: Hari Sekhon
Improve Stochastic Balancer to include the HDFS region location hints to avoid
HDFS Balancer destroying data locality.
Right now according to a mix of docs, jiras and mailing list info it appears
that one must change
{code:java}
hbase.master.loadbalancer.class{code}
to the org.apache.hadoop.hbase.favored.FavoredNodeLoadBalancer as it looks like
this functionality is only within FavouredNodeBalancer and not the standard
Stochastic Balancer.
[http://hbase.apache.org/book.html#_hbase_and_hdfs]
This is not ideal because we'd still like to use all the heuristics and work
that has gone in the Stochastic Balancer which I believe right now is the best
and most mature HBase balancer.
See also the linked Jiras and this discussion:
[http://apache-hbase.679495.n3.nabble.com/HDFS-Balancer-td4086607.html]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[
https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570040#comment-16570040
]
Reid Chan commented on HBASE-18201:
---
{quote}2. Can check other available Compression.Algorithm as well? (not only
GZ).{quote}
Just ignore it, it is not part of this jira.
> add UT and docs for DataBlockEncodingTool
> -
>
> Key: HBASE-18201
> URL: https://issues.apache.org/jira/browse/HBASE-18201
> Project: HBase
> Issue Type: Sub-task
> Components: tooling
>Reporter: Chia-Ping Tsai
>Assignee: Kuan-Po Tseng
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-18201.master.001.patch,
> HBASE-18201.master.002.patch, HBASE-18201.master.002.patch,
> HBASE-18201.master.003.patch, HBASE-18201.master.004.patch,
> HBASE-18201.master.005.patch, HBASE-18201.master.005.patch,
> HBASE-18201.master.005.patch
>
>
> There is no example, documents, or tests for DataBlockEncodingTool. We should
> have it friendly if any use case exists. Otherwise, we should just get rid of
> it because DataBlockEncodingTool presumes that the implementation of cell
> returned from DataBlockEncoder is KeyValue. The presume may obstruct the
> cleanup of KeyValue references in the code base of read/write path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570012#comment-16570012
]
Ted Yu edited comment on HBASE-19008 at 8/6/18 10:33 AM:
-
Please fix the following compilation errors:
{code}
[ERROR]
/testptch/hbase/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java:[413,24]
[ArrayHashCode] hashcode method on array does not hash array contents
(see http://errorprone.info/bugpattern/ArrayHashCode)
Did you mean 'return
Objects.hash(this.prevFilterRCList,this.prevCellList,Arrays.hashCode(this.filters.toArray()));'?
[ERROR]
/testptch/hbase/hbase-spark/src/main/java/org/apache/hadoop/hbase/spark/SparkSQLPushDownFilter.java:[307,33]
[ArrayHashCode] hashcode method on array does not hash array contents
(see http://errorprone.info/bugpattern/ArrayHashCode)
Did you mean 'hash=hash+Arrays.hashCode(val);'?
{code}
Please also address findbugs and checkstyle warnings.
The above comment was made before seeing patch v5.
was (Author: yuzhih...@gmail.com):
Please fix the following compilation errors:
{code}
[ERROR]
/testptch/hbase/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java:[413,24]
[ArrayHashCode] hashcode method on array does not hash array contents
(see http://errorprone.info/bugpattern/ArrayHashCode)
Did you mean 'return
Objects.hash(this.prevFilterRCList,this.prevCellList,Arrays.hashCode(this.filters.toArray()));'?
[ERROR]
/testptch/hbase/hbase-spark/src/main/java/org/apache/hadoop/hbase/spark/SparkSQLPushDownFilter.java:[307,33]
[ArrayHashCode] hashcode method on array does not hash array contents
(see http://errorprone.info/bugpattern/ArrayHashCode)
Did you mean 'hash=hash+Arrays.hashCode(val);'?
{code}
Please also address findbugs and checkstyle warnings.
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-2.patch,
> HBASE-19008-3.patch, HBASE-19008-4.patch, HBASE-19008-5.patch,
> HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570012#comment-16570012
]
Ted Yu commented on HBASE-19008:
Please fix the following compilation errors:
{code}
[ERROR]
/testptch/hbase/hbase-client/src/main/java/org/apache/hadoop/hbase/filter/FilterListWithOR.java:[413,24]
[ArrayHashCode] hashcode method on array does not hash array contents
(see http://errorprone.info/bugpattern/ArrayHashCode)
Did you mean 'return
Objects.hash(this.prevFilterRCList,this.prevCellList,Arrays.hashCode(this.filters.toArray()));'?
[ERROR]
/testptch/hbase/hbase-spark/src/main/java/org/apache/hadoop/hbase/spark/SparkSQLPushDownFilter.java:[307,33]
[ArrayHashCode] hashcode method on array does not hash array contents
(see http://errorprone.info/bugpattern/ArrayHashCode)
Did you mean 'hash=hash+Arrays.hashCode(val);'?
{code}
Please also address findbugs and checkstyle warnings.
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-2.patch,
> HBASE-19008-3.patch, HBASE-19008-4.patch, HBASE-19008-5.patch,
> HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21006) Balancer - data locality drops 30-40% across all nodes after every cluster-wide rolling restart, not migrating regions back to original RegionServers?
[ https://issues.apache.org/jira/browse/HBASE-21006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sekhon updated HBASE-21006: Issue Type: Bug (was: Improvement) > Balancer - data locality drops 30-40% across all nodes after every > cluster-wide rolling restart, not migrating regions back to original > RegionServers? > -- > > Key: HBASE-21006 > URL: https://issues.apache.org/jira/browse/HBASE-21006 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 1.1.2 > Environment: HDP 2.6 >Reporter: Hari Sekhon >Priority: Major > > After doing rolling restarts of my HBase cluster the data locality drops by > 30-40% every time which implies the stochastic balancer is not optimizing for > data locality enough, at least not under the circumstance of rolling > restarts, and that it must not be balancing the regions back to their > original RegionServers. > The stochastic balancer is supposed to take data locality in to account but > if this is the case, surely it should move regions back to their original > RegionServers and data locality should return back to around where it was, > not drop by 30-40% percent every time I need to do some tuning and a rolling > restart. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569932#comment-16569932 ] Reid Chan commented on HBASE-18201: --- I think we need one more vote to let this go, since i'm not familiar to this part, there may be somewhere i missed. Ping [~chia7712], [~stack] to take a look. > add UT and docs for DataBlockEncodingTool > - > > Key: HBASE-18201 > URL: https://issues.apache.org/jira/browse/HBASE-18201 > Project: HBase > Issue Type: Sub-task > Components: tooling >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng >Priority: Minor > Labels: beginner > Attachments: HBASE-18201.master.001.patch, > HBASE-18201.master.002.patch, HBASE-18201.master.002.patch, > HBASE-18201.master.003.patch, HBASE-18201.master.004.patch, > HBASE-18201.master.005.patch, HBASE-18201.master.005.patch, > HBASE-18201.master.005.patch > > > There is no example, documents, or tests for DataBlockEncodingTool. We should > have it friendly if any use case exists. Otherwise, we should just get rid of > it because DataBlockEncodingTool presumes that the implementation of cell > returned from DataBlockEncoder is KeyValue. The presume may obstruct the > cleanup of KeyValue references in the code base of read/write path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[
https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569931#comment-16569931
]
Reid Chan commented on HBASE-18201:
---
lgtm, overall.
You may need rebase again, i tried {{git apply}} with some conflicts (docs
parts).
Nit.
Add a blank new line between
{code}
70 + private List isTagsLenZero = new ArrayList<>();
71/**
{code}
{code} //check if HFile use Tag. {code}
blank between `//` and `check`, and pay attention to properly use upper case,
like `C` in this case. There're few other places, please check.
Naming: {code}itIsTagsLenZero > it // or iterator is better{code}
Question:
1. Is {{meta.getDataBlockEncoding()}} equals {{this.encoding}} (it looks so to
me). If so, {{this.encoding = encoding}} can be removed IMO.
2. Can check other available Compression.Algorithm as well? (not only GZ).
> add UT and docs for DataBlockEncodingTool
> -
>
> Key: HBASE-18201
> URL: https://issues.apache.org/jira/browse/HBASE-18201
> Project: HBase
> Issue Type: Sub-task
> Components: tooling
>Reporter: Chia-Ping Tsai
>Assignee: Kuan-Po Tseng
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-18201.master.001.patch,
> HBASE-18201.master.002.patch, HBASE-18201.master.002.patch,
> HBASE-18201.master.003.patch, HBASE-18201.master.004.patch,
> HBASE-18201.master.005.patch, HBASE-18201.master.005.patch,
> HBASE-18201.master.005.patch
>
>
> There is no example, documents, or tests for DataBlockEncodingTool. We should
> have it friendly if any use case exists. Otherwise, we should just get rid of
> it because DataBlockEncodingTool presumes that the implementation of cell
> returned from DataBlockEncoder is KeyValue. The presume may obstruct the
> cleanup of KeyValue references in the code base of read/write path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20881) Introduce a region transition procedure to handle all the state transition for a region
[ https://issues.apache.org/jira/browse/HBASE-20881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20881: -- Attachment: HBASE-20881-v2.patch > Introduce a region transition procedure to handle all the state transition > for a region > --- > > Key: HBASE-20881 > URL: https://issues.apache.org/jira/browse/HBASE-20881 > Project: HBase > Issue Type: Sub-task > Components: amv2, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-20881-v1.patch, HBASE-20881-v2.patch, > HBASE-20881.patch > > > Now have an AssignProcedure, an UnssignProcedure, and also a > MoveRegionProcedure which schedules an AssignProcedure and an > UnssignProcedure to move a region. This makes the logic a bit complicated, as > MRP is not a RIT, so when SCP can not interrupt it directly... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21012) Revert the change of serializing TimeRangeTracker
[ https://issues.apache.org/jira/browse/HBASE-21012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-21012: --- Summary: Revert the change of serializing TimeRangeTracker (was: Revert the use of proto.TimeRangeTracker when writing hfile) > Revert the change of serializing TimeRangeTracker > - > > Key: HBASE-21012 > URL: https://issues.apache.org/jira/browse/HBASE-21012 > Project: HBase > Issue Type: Sub-task >Reporter: Chia-Ping Tsai >Priority: Critical > Fix For: 3.0.0, 2.0.2, 2.1.1 > > > HBASE-18754 change the serialization of TimeRangeTracker from "manual way" to > protobuf. However, the change breaks the backward compatibility of hfile. We > should revert the change ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen updated HBASE-19008:
Attachment: HBASE-19008-5.patch
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-2.patch,
> HBASE-19008-3.patch, HBASE-19008-4.patch, HBASE-19008-5.patch,
> HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
