[jira] [Created] (HBASE-21516) Use AsyncConnection instead of Connection in SecureBulkLoadManager
Duo Zhang created HBASE-21516: - Summary: Use AsyncConnection instead of Connection in SecureBulkLoadManager Key: HBASE-21516 URL: https://issues.apache.org/jira/browse/HBASE-21516 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21507: -- Fix Version/s: 2.0.4 2.1.2 2.2.0 > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: lixiaobao >Assignee: lixiaobao >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.2, 2.0.4 > > Attachments: HBASE-21506-v1.patch, hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-21507: - Assignee: lixiaobao > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: lixiaobao >Assignee: lixiaobao >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21506-v1.patch, hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21507: -- Attachment: HBASE-21506-v1.patch > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: lixiaobao >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21506-v1.patch, hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21515: -- Attachment: HBASE-21515-v1.patch > Also initialize an AsyncClusterConnection in HRegionServer > -- > > Key: HBASE-21515 > URL: https://issues.apache.org/jira/browse/HBASE-21515 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21515-v1.patch, HBASE-21515.patch > > > Plan to do this incrementally, so first we will only introduce the new > AsyncClusterConnection, without deleting the old ClusterConnection. And we > can move the code which uses the ClusterConnection to use > AsyncClusterConnection, through different sub tasks here. And finally we can > remove the ClusterConnection completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21401) Sanity check in BaseDecoder#parseCell
[ https://issues.apache.org/jira/browse/HBASE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1667#comment-1667 ] Zheng Hu commented on HBASE-21401: -- bq. Could the validity checks be done inline as we construct the KeyValue or the NoTagsKeyValue instead? Thanks. The current patch will validate the bytes when deserialize from socket or from WAL.. but the HFIleReaderImpl won't use the KeyValueUtil.createKeyValueFromInputStream to construte the KeyValue, so it will escape the check. I think it's OK to move the validating when we construct the KeyValue if the JMH method testing (exclude the impact from JIT) also says the checkKeyValueBytes won't cost too much time, I'll update the JMH test result in HBASE-21459. Thanks. > Sanity check in BaseDecoder#parseCell > - > > Key: HBASE-21401 > URL: https://issues.apache.org/jira/browse/HBASE-21401 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2 > > Attachments: HBASE-21401.v1.patch, HBASE-21401.v2.patch, > HBASE-21401.v3.patch, HBASE-21401.v4.patch, HBASE-21401.v4.patch, > HBASE-21401.v5.patch > > > In KeyValueDecoder & ByteBuffKeyValueDecoder, we pass a byte buffer to > initialize the Cell without a sanity check (check each field's offset > exceed the byte buffer or not), so ArrayIndexOutOfBoundsException may happen > when read the cell's fields, such as HBASE-21379, it's hard to debug this > kind of bug. > An earlier check will help to find such kind of bugs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21510) Test TestRegisterPeerWorkerWhenRestarting is flakey
[
https://issues.apache.org/jira/browse/HBASE-21510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21510:
--
Attachment: HBASE-21510-addendum.patch
> Test TestRegisterPeerWorkerWhenRestarting is flakey
> ---
>
> Key: HBASE-21510
> URL: https://issues.apache.org/jira/browse/HBASE-21510
> Project: HBase
> Issue Type: Bug
> Components: Replication, test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21510-addendum.patch, HBASE-21510.patch
>
>
> {noformat}
> 2018-11-22 16:57:34,066 WARN [Thread-1101]
> client.HBaseAdmin$ProcedureFuture(3528): failed to get the procedure result
> procId=26
> org.apache.hadoop.hbase.DoNotRetryIOException: Unable to instantiate
> exception received from
> server:org.apache.hadoop.hbase.master.HMaster$MasterStoppedException.(java.lang.String)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:93)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:361)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:349)
> at
> org.apache.hadoop.hbase.client.MasterCallable.call(MasterCallable.java:101)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3133)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3125)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.access$700(HBaseAdmin.java:234)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.getProcedureResult(HBaseAdmin.java:3571)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:3523)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:3479)
> at org.apache.hadoop.hbase.client.HBaseAdmin.get(HBaseAdmin.java:2199)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.transitReplicationPeerSyncReplicationState(HBaseAdmin.java:4073)
> at
> org.apache.hadoop.hbase.master.replication.TestRegisterPeerWorkerWhenRestarting$1.run(TestRegisterPeerWorkerWhenRestarting.java:102)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.master.HMaster$MasterStoppedException):
> org.apache.hadoop.hbase.master.HMaster$MasterStoppedException
> at
> org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:3080)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.getProcedureResult(MasterRpcServices.java:1181)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:387)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
> at
> org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
> at
>
[jira] [Commented] (HBASE-18735) Provide a fast mechanism for shutting down mini cluster
[ https://issues.apache.org/jira/browse/HBASE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1666#comment-1666 ] Anoop Sam John commented on HBASE-18735: +1 for the idea. > Provide a fast mechanism for shutting down mini cluster > --- > > Key: HBASE-18735 > URL: https://issues.apache.org/jira/browse/HBASE-18735 > Project: HBase > Issue Type: Wish >Reporter: Samarth Jain >Assignee: Artem Ervits >Priority: Major > > The current mechanism of shutting down a mini cluster through > HBaseTestingUtility.shutDownMiniCluster can take a lot of time when the mini > cluster almost has a lot of tables. A lot of this time is spent in closing > all the user regions. It would be nice to have a mechanism where this > shutdown can happen quickly without having to worry about closing these user > regions. At the same time, this mechanism would need to make sure that all > the critical system resources like file handles and network ports are still > released so that subsequently initialized mini clusters on the same JVM or > system won't run into resource issues. This would make testing using HBase > mini clusters much faster and immensely help out test frameworks of dependent > projects like Phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699988#comment-16699988 ] Anoop Sam John commented on HBASE-21507: This needs to be fixed in all 2.x versions as well.. Hope the patch will apply cleanly on all branches. > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: lixiaobao >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-21507: --- Priority: Major (was: Minor) > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: lixiaobao >Priority: Major > Fix For: 3.0.0 > > Attachments: hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-21507: --- Affects Version/s: (was: 2.1.1) 2.0.0 > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: lixiaobao >Priority: Minor > Fix For: 3.0.0 > > Attachments: hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21507) Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method
[ https://issues.apache.org/jira/browse/HBASE-21507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699986#comment-16699986 ] Anoop Sam John commented on HBASE-21507: Checking DateTieredMultiFileWriter, yes seems the writer can be null.. +1 for the patch.. > Compaction failed when execute AbstractMultiFileWriter.beforeShipped() method > - > > Key: HBASE-21507 > URL: https://issues.apache.org/jira/browse/HBASE-21507 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.1.1 >Reporter: lixiaobao >Priority: Minor > Fix For: 3.0.0 > > Attachments: hbase-21506.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > 2018-11-23 15:36:20,176 ERROR > [regionserver/ml42:16020-longCompactions-1542957876702] > regionserver.CompactSplit: Compaction failed > region=test_db_zero9,4071\x009223370493980145710\x001,1542958557994.3b1f0283a8d07a2469e5e87f87d827da., > storeName=cf, priority=59, startTime=1542958579969 > java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.AbstractMultiFileWriter.beforeShipped(AbstractMultiFileWriter.java:124) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:432) > at > org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:327) > at > org.apache.hadoop.hbase.regionserver.compactions.DateTieredCompactor.compact(DateTieredCompactor.java:64) > at > org.apache.hadoop.hbase.regionserver.DateTieredStoreEngine$DateTieredCompactionContext.compact(DateTieredStoreEngine.java:96) > at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) > at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) > at > org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21498) Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a new BlockCache
[ https://issues.apache.org/jira/browse/HBASE-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21498: --- Resolution: Fixed Fix Version/s: 2.2.0 3.0.0 Status: Resolved (was: Patch Available) Failed ut not related. Pushed to master and branch-2. Thanks all for reviewing. > Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a > new BlockCache > -- > > Key: HBASE-21498 > URL: https://issues.apache.org/jira/browse/HBASE-21498 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21498.master.001.patch, > HBASE-21498.master.002.patch, HBASE-21498.master.003.patch, > HBASE-21498.master.004.patch, HBASE-21498.master.005.patch, > HBASE-21498.master.006.patch, HBASE-21498.master.006.patch, > HBASE-21498.master.007.patch, HBASE-21498.master.007.patch > > > In our cluster, we use a small heap/offheap config for master. After > HBASE-21290, master doesn't instantiate BlockCache when it not carry table. > But it will new CacheConfig in SplitTableRegionProcedure.splitStoreFiles > method. And it will instantiate a new BlockCache if it not initialized before > and make master OOM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699984#comment-16699984
]
Duo Zhang commented on HBASE-21154:
---
It is synchronous replication, not serial replication :)
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21300) Fix the wrong reference file path when restoring snapshots for tables with MOB columns
[
https://issues.apache.org/jira/browse/HBASE-21300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Mei updated HBASE-21300:
---
Attachment: HBASE-21300.master.006.patch
> Fix the wrong reference file path when restoring snapshots for tables with
> MOB columns
> --
>
> Key: HBASE-21300
> URL: https://issues.apache.org/jira/browse/HBASE-21300
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21300.master.001.patch,
> HBASE-21300.master.002.patch, HBASE-21300.master.003.patch,
> HBASE-21300.master.004.patch, HBASE-21300.master.005.patch,
> HBASE-21300.master.006.patch
>
>
> When restoring snapshots for tables with MOB columns, the reference files for
> mob region are created under hbase root dir, rather than restore dir.
> Some of the mob reference file paths are as follows:
> {quote}hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A/d41d8cd98f00b204e9800998ecf8427e201810120fc8e2446f174598a7280a81b1134cee
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A/ns_testMob=testMob=057a856eb65753c6e6bdb168ba58a0b2-d41d8cd98f00b204e9800998ecf8427e201810120fc8e2446f174598a7280a81b1134cee
> {quote}
> The restore dir files are as follows:
> {quote}hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/.regioninfo
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/A
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/A/ns_testMob=testMob=ecdf66f0d8c09a816faf37336ad262e1-7208172df03b46518370643aa28ffd05
> {quote}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21401) Sanity check in BaseDecoder#parseCell
[ https://issues.apache.org/jira/browse/HBASE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699975#comment-16699975 ] stack commented on HBASE-21401: --- Thanks for removing from Master until we figure this. Its a total parse of the Key ... and if tags, the KV... doing deserialization of offsets and lengths to walk the Key only to throw away these deserializations. Could the validity checks be done inline as we construct the KeyValue or the NoTagsKeyValue instead? Thanks. > Sanity check in BaseDecoder#parseCell > - > > Key: HBASE-21401 > URL: https://issues.apache.org/jira/browse/HBASE-21401 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2 > > Attachments: HBASE-21401.v1.patch, HBASE-21401.v2.patch, > HBASE-21401.v3.patch, HBASE-21401.v4.patch, HBASE-21401.v4.patch, > HBASE-21401.v5.patch > > > In KeyValueDecoder & ByteBuffKeyValueDecoder, we pass a byte buffer to > initialize the Cell without a sanity check (check each field's offset > exceed the byte buffer or not), so ArrayIndexOutOfBoundsException may happen > when read the cell's fields, such as HBASE-21379, it's hard to debug this > kind of bug. > An earlier check will help to find such kind of bugs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699976#comment-16699976
]
stack commented on HBASE-21154:
---
bq. For me I think it should be 3.0.0 only, as this is a big change, although
we suppose that user should not access the namespace table directly...
But no serial replication w/o this change?
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699971#comment-16699971 ] Duo Zhang commented on HBASE-21515: --- Let me fix the failed UT first. Plan to do this in a feature branch. > Also initialize an AsyncClusterConnection in HRegionServer > -- > > Key: HBASE-21515 > URL: https://issues.apache.org/jira/browse/HBASE-21515 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21515.patch > > > Plan to do this incrementally, so first we will only introduce the new > AsyncClusterConnection, without deleting the old ClusterConnection. And we > can move the code which uses the ClusterConnection to use > AsyncClusterConnection, through different sub tasks here. And finally we can > remove the ClusterConnection completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699965#comment-16699965
]
Duo Zhang commented on HBASE-21154:
---
For me I think it should be 3.0.0 only, as this is a big change, although we
suppose that user should not access the namespace table directly...
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21447) HBCK2 tool have questions on holes when HBCK2 checks region chain
[
https://issues.apache.org/jira/browse/HBASE-21447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699960#comment-16699960
]
stack commented on HBASE-21447:
---
[~nicholasjiang] How did you do it? Eyeballing meta? Were some regions split or
offlined? Was the table scannable/readable? Thanks.
> HBCK2 tool have questions on holes when HBCK2 checks region chain
> ---
>
> Key: HBASE-21447
> URL: https://issues.apache.org/jira/browse/HBASE-21447
> Project: HBase
> Issue Type: Improvement
> Components: hbck2
>Affects Versions: 2.0.2
>Reporter: Nicholas Jiang
>Priority: Major
> Attachments: Hole.png
>
>
> [hbck2]https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2
> This HBCK2 tool have some questions on holes when HBCK2 checks region chain
> as follows.
> {code:java}
> ERROR: There is a hole in the region chain between \x01F\x00\x00 and
> \x02\x8C\x00\x00. You need to create a new .regioninfo and region dir in hdfs
> to plug the hole.
> ERROR: There is a hole in the region chain between \x05\x18\x00\x00 and
> \x06^\x00\x00. You need to create a new .regioninfo and region dir in hdfs to
> plug the hole.
> ERROR: There is a hole in the region chain between \x07\x01\x00\x00 and
> \x07\xA4\x00\x00. You need to create a new .regioninfo and region dir in hdfs
> to plug the hole.
> ERROR: There is a hole in the region chain between \x08G\x00\x00 and
> \x09\x8D\x00\x00. You need to create a new .regioninfo and region dir in hdfs
> to plug the hole.
> ERROR: There is a hole in the region chain between \x0A0\x00\x00 and
> \x0Bv\x00\x00. You need to create a new .regioninfo and region dir in hdfs to
> plug the hole.
> ERROR: There is a hole in the region chain between \x0C\x19\x00\x00 and
> \x0C\xBC\x00\x00. You need to create a new .regioninfo and region dir in hdfs
> to plug the hole.
> ERROR: There is a hole in the region chain between \x0D_\x00\x00 and
> \x0E\xA5\x00\x00. You need to create a new .regioninfo and region dir in hdfs
> to plug the hole.
> ERROR: There is a hole in the region chain between \x0F\xEB\x00\x00 and
> \x111\x00\x00. You need to create a new .regioninfo and region dir in hdfs to
> plug the hole.
> ERROR: There is a hole in the region chain between \x16I\x00\x00 and
> \x16\xEC\x00\x00. You need to create a new .regioninfo and region dir in hdfs
> to plug the hole.
> ERROR: There is a hole in the region chain between (\xC0\x00\x00 and
> *\x06\x00\x00. You need to create a new .regioninfo and region dir in hdfs to
> plug the hole.
> {code}
> !Hole.png!
> This hole problem can't be solved by HBCK2 tool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699942#comment-16699942
]
stack commented on HBASE-21492:
---
Patch LGTM. Is test failure related? Let me rerun the patch so we can tell when
[~belugabehr] wakes up
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch,
> HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699953#comment-16699953
]
stack commented on HBASE-21154:
---
Is this a 3.0.0 only fix? We don't want this in 2.2.0? Or, should we do 3.0.0
rather than a 2.2.0? Can chat up on dev-list
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699951#comment-16699951 ] stack commented on HBASE-21515: --- Easier to review on rb. > Also initialize an AsyncClusterConnection in HRegionServer > -- > > Key: HBASE-21515 > URL: https://issues.apache.org/jira/browse/HBASE-21515 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21515.patch > > > Plan to do this incrementally, so first we will only introduce the new > AsyncClusterConnection, without deleting the old ClusterConnection. And we > can move the code which uses the ClusterConnection to use > AsyncClusterConnection, through different sub tasks here. And finally we can > remove the ClusterConnection completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21492:
--
Attachment: HBASE-21492.2.patch
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch,
> HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21300) Fix the wrong reference file path when restoring snapshots for tables with MOB columns
[
https://issues.apache.org/jira/browse/HBASE-21300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699944#comment-16699944
]
Guanghao Zhang commented on HBASE-21300:
+1. Please fix the checkstyle report [~Yi Mei]
> Fix the wrong reference file path when restoring snapshots for tables with
> MOB columns
> --
>
> Key: HBASE-21300
> URL: https://issues.apache.org/jira/browse/HBASE-21300
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21300.master.001.patch,
> HBASE-21300.master.002.patch, HBASE-21300.master.003.patch,
> HBASE-21300.master.004.patch, HBASE-21300.master.005.patch
>
>
> When restoring snapshots for tables with MOB columns, the reference files for
> mob region are created under hbase root dir, rather than restore dir.
> Some of the mob reference file paths are as follows:
> {quote}hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A/d41d8cd98f00b204e9800998ecf8427e201810120fc8e2446f174598a7280a81b1134cee
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A/ns_testMob=testMob=057a856eb65753c6e6bdb168ba58a0b2-d41d8cd98f00b204e9800998ecf8427e201810120fc8e2446f174598a7280a81b1134cee
> {quote}
> The restore dir files are as follows:
> {quote}hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/.regioninfo
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/A
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/A/ns_testMob=testMob=ecdf66f0d8c09a816faf37336ad262e1-7208172df03b46518370643aa28ffd05
> {quote}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21300) Fix the wrong reference file path when restoring snapshots for tables with MOB columns
[
https://issues.apache.org/jira/browse/HBASE-21300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699921#comment-16699921
]
Hadoop QA commented on HBASE-21300:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 1 new + 42 unchanged
- 0 fixed = 43 total (was 42) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
59s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
17s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}168m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21300 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949593/HBASE-21300.master.005.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6ca386686b9e 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1acbd36c90 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/15123/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15123/testReport/ |
| Max. process+thread count | 4906 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21509) /hbase/WALs directory has large number of hlogs, which cannot be deleted correctly
[ https://issues.apache.org/jira/browse/HBASE-21509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699919#comment-16699919 ] Allan Yang commented on HBASE-21509: I'm not fully understand why 'getMetaRegionLocation() returns null value， it may cause meta table lose any data'. Can you describe more or provide a UT? > /hbase/WALs directory has large number of hlogs, which cannot be deleted > correctly > -- > > Key: HBASE-21509 > URL: https://issues.apache.org/jira/browse/HBASE-21509 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Bo Cui >Priority: Minor > > When HMaster is initializing, if getMetaRegionLocation() returns null value, > then some wal(including metaWAL) cannot be deleted. > for example > before restarts > /hbase/WALs/10-10-10-129,21302,1543048601526/10-10-10-129%2C21302%2C1543048601526.meta.1543048613941.meta > after restarts > /hbase/WALs/10-10-10-129,21302,1543048601526-splitting/10-10-10-129%2C21302%2C1543048601526.meta.1543048613941.meta > /hbase/WALs/10-10-10-29,21302,1543048867527/10-10-10-29%2C21302%2C1543048867527.meta.1543048907265.meta -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21514) Refactor CacheConfig
[ https://issues.apache.org/jira/browse/HBASE-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-21514: --- Fix Version/s: 3.0.0 > Refactor CacheConfig > > > Key: HBASE-21514 > URL: https://issues.apache.org/jira/browse/HBASE-21514 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > > One basic idea is move the global cache instances from CacheConfig. Only keep > config stuff in CacheConfig. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21481) [acl] Superuser's permissions should not be granted or revoked by any non-su global admin
[
https://issues.apache.org/jira/browse/HBASE-21481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699883#comment-16699883
]
Reid Chan edited comment on HBASE-21481 at 11/27/18 4:40 AM:
-
It's a non-trivial patch i think, need +1 from team member.
Would you mind spending some time to review this patch? [~tedyu] [~elserj]
[~busbey] [~dbist13] [~psomogyi] (after Thanksgiving holiday, of course.
Many thanks.
was (Author: reidchan):
It's a non-trivial patch i think, need +1 from team member.
Would you mind spending some time to review this patch? [~tedyu] [~elserj]
[~busbey] [~dbist13] [~psomogyi]
Many thanks.
> [acl] Superuser's permissions should not be granted or revoked by any non-su
> global admin
> -
>
> Key: HBASE-21481
> URL: https://issues.apache.org/jira/browse/HBASE-21481
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Labels: ACL, security-issue
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21481.master.001.patch,
> HBASE-21481.master.002.patch, HBASE-21481.master.003.patch,
> HBASE-21481.master.004.patch, HBASE-21481.master.005.patch
>
>
> Superusers are {{hbase.superuser}} listed in configuration and plus the one
> who start master process, these two may be overlap.
> A superuser must be a global admin, but a global admin may not be a
> superuser, possibly granted afterwards.
> For now, an non-su global admin with a Global.ADMIN permission can grant or
> revoke any superuser's permission, accidentally or deliberately.
> The purpose of this issue is to ban this action.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21514) Refactor CacheConfig
[ https://issues.apache.org/jira/browse/HBASE-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699915#comment-16699915 ] Anoop Sam John commented on HBASE-21514: When we have a test Mini cluster with 1+ RSs, ideally every RS instance should have its own single instance of Block Cache. But right now what happens is every RS in mini cluster will share one BC because the BC is kept as Static instance in CacheConfig and Minicluster is on single JVM. As said above CacheConfig should have config and other related stuff alone. The BC instance should be some thing kept at HRS level and instance level single object. Similar issue might be there with MSLAB pool and chunks also. After this is been fixed, we will reactor that area also. > Refactor CacheConfig > > > Key: HBASE-21514 > URL: https://issues.apache.org/jira/browse/HBASE-21514 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > > One basic idea is move the global cache instances from CacheConfig. Only keep > config stuff in CacheConfig. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699889#comment-16699889
]
Hadoop QA commented on HBASE-21492:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} hbase-server: The patch generated 0 new + 8
unchanged - 1 fixed = 8 total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch passed checkstyle in hbase-mapreduce
{color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 27s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}133m 2s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
14s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestFailedAppendAndSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21492 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949578/HBASE-21492.2.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c5d54e225c8d 4.4.0-131-generic #157~14.04.1-Ubuntu SMP Fri Jul
13 08:53:17 UTC 2018 x86_64 GNU/Linux |
[jira] [Commented] (HBASE-21481) [acl] Superuser's permissions should not be granted or revoked by any non-su global admin
[
https://issues.apache.org/jira/browse/HBASE-21481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699883#comment-16699883
]
Reid Chan commented on HBASE-21481:
---
It's a non-trivial patch i think, need +1 from team member.
Would you mind spending some time to review this patch? [~tedyu] [~elserj]
[~busbey] [~dbist13] [~psomogyi]
Many thanks.
> [acl] Superuser's permissions should not be granted or revoked by any non-su
> global admin
> -
>
> Key: HBASE-21481
> URL: https://issues.apache.org/jira/browse/HBASE-21481
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Labels: ACL, security-issue
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21481.master.001.patch,
> HBASE-21481.master.002.patch, HBASE-21481.master.003.patch,
> HBASE-21481.master.004.patch, HBASE-21481.master.005.patch
>
>
> Superusers are {{hbase.superuser}} listed in configuration and plus the one
> who start master process, these two may be overlap.
> A superuser must be a global admin, but a global admin may not be a
> superuser, possibly granted afterwards.
> For now, an non-su global admin with a Global.ADMIN permission can grant or
> revoke any superuser's permission, accidentally or deliberately.
> The purpose of this issue is to ban this action.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699882#comment-16699882
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #55 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/55/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/55//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/55//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/55//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
[
https://issues.apache.org/jira/browse/HBASE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699864#comment-16699864
]
Hadoop QA commented on HBASE-21515:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} hbase-client: The patch generated 1 new + 0 unchanged
- 3 fixed = 1 total (was 3) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
10s{color} | {color:red} hbase-server: The patch generated 6 new + 286
unchanged - 2 fixed = 292 total (was 288) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 19s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red} hbase-client generated 1 new + 2 unchanged - 0 fixed =
3 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
16s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 4s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestClientClusterMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949592/HBASE-21515.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1d68d53622c3 4.4.0-138-generic #164-Ubuntu SMP
[jira] [Updated] (HBASE-21300) Fix the wrong reference file path when restoring snapshots for tables with MOB columns
[
https://issues.apache.org/jira/browse/HBASE-21300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Mei updated HBASE-21300:
---
Attachment: HBASE-21300.master.005.patch
> Fix the wrong reference file path when restoring snapshots for tables with
> MOB columns
> --
>
> Key: HBASE-21300
> URL: https://issues.apache.org/jira/browse/HBASE-21300
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21300.master.001.patch,
> HBASE-21300.master.002.patch, HBASE-21300.master.003.patch,
> HBASE-21300.master.004.patch, HBASE-21300.master.005.patch
>
>
> When restoring snapshots for tables with MOB columns, the reference files for
> mob region are created under hbase root dir, rather than restore dir.
> Some of the mob reference file paths are as follows:
> {quote}hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A/d41d8cd98f00b204e9800998ecf8427e201810120fc8e2446f174598a7280a81b1134cee
> hdfs:/7ae0d109-3ca4-d0e7-7250-62ed234ab247/mobdir/data/ns_testMob/testMob/057a856eb65753c6e6bdb168ba58a0b2/A/ns_testMob=testMob=057a856eb65753c6e6bdb168ba58a0b2-d41d8cd98f00b204e9800998ecf8427e201810120fc8e2446f174598a7280a81b1134cee
> {quote}
> The restore dir files are as follows:
> {quote}hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/.regioninfo
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/A
> hdfs://hbase/.tmpdir-to-restore-snapshot/856e06fa-e018-4e95-9647-2cfbd5161e7e/data/ns_testMob/testMob/ecdf66f0d8c09a816faf37336ad262e1/A/ns_testMob=testMob=ecdf66f0d8c09a816faf37336ad262e1-7208172df03b46518370643aa28ffd05
> {quote}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21515: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > Also initialize an AsyncClusterConnection in HRegionServer > -- > > Key: HBASE-21515 > URL: https://issues.apache.org/jira/browse/HBASE-21515 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21515.patch > > > Plan to do this incrementally, so first we will only introduce the new > AsyncClusterConnection, without deleting the old ClusterConnection. And we > can move the code which uses the ClusterConnection to use > AsyncClusterConnection, through different sub tasks here. And finally we can > remove the ClusterConnection completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-21515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21515: -- Attachment: HBASE-21515.patch > Also initialize an AsyncClusterConnection in HRegionServer > -- > > Key: HBASE-21515 > URL: https://issues.apache.org/jira/browse/HBASE-21515 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Duo Zhang >Priority: Major > Attachments: HBASE-21515.patch > > > Plan to do this incrementally, so first we will only introduce the new > AsyncClusterConnection, without deleting the old ClusterConnection. And we > can move the code which uses the ClusterConnection to use > AsyncClusterConnection, through different sub tasks here. And finally we can > remove the ClusterConnection completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21509) /hbase/WALs directory has large number of hlogs, which cannot be deleted correctly
[ https://issues.apache.org/jira/browse/HBASE-21509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699797#comment-16699797 ] Bo Cui commented on HBASE-21509: [~allan163] right！ u talking about is HBase-14223， yes，we also encountered this issue but if getMetaRegionLocation() returns null value， it may cause meta table lose any data. > /hbase/WALs directory has large number of hlogs, which cannot be deleted > correctly > -- > > Key: HBASE-21509 > URL: https://issues.apache.org/jira/browse/HBASE-21509 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Bo Cui >Priority: Minor > > When HMaster is initializing, if getMetaRegionLocation() returns null value, > then some wal(including metaWAL) cannot be deleted. > for example > before restarts > /hbase/WALs/10-10-10-129,21302,1543048601526/10-10-10-129%2C21302%2C1543048601526.meta.1543048613941.meta > after restarts > /hbase/WALs/10-10-10-129,21302,1543048601526-splitting/10-10-10-129%2C21302%2C1543048601526.meta.1543048613941.meta > /hbase/WALs/10-10-10-29,21302,1543048867527/10-10-10-29%2C21302%2C1543048867527.meta.1543048907265.meta -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21498) Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a new BlockCache
[
https://issues.apache.org/jira/browse/HBASE-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699796#comment-16699796
]
Guanghao Zhang commented on HBASE-21498:
{quote}bq.Fine lets do the bigger change of CacheConfig refactor etc in a new
issue. Lets fix the bigger issue fix . Can u pls raise the new one.
{quote}
Open HBASE-21514 to refactor CacheConfig.
> Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a
> new BlockCache
> --
>
> Key: HBASE-21498
> URL: https://issues.apache.org/jira/browse/HBASE-21498
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-21498.master.001.patch,
> HBASE-21498.master.002.patch, HBASE-21498.master.003.patch,
> HBASE-21498.master.004.patch, HBASE-21498.master.005.patch,
> HBASE-21498.master.006.patch, HBASE-21498.master.006.patch,
> HBASE-21498.master.007.patch, HBASE-21498.master.007.patch
>
>
> In our cluster, we use a small heap/offheap config for master. After
> HBASE-21290, master doesn't instantiate BlockCache when it not carry table.
> But it will new CacheConfig in SplitTableRegionProcedure.splitStoreFiles
> method. And it will instantiate a new BlockCache if it not initialized before
> and make master OOM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21515) Also initialize an AsyncClusterConnection in HRegionServer
Duo Zhang created HBASE-21515: - Summary: Also initialize an AsyncClusterConnection in HRegionServer Key: HBASE-21515 URL: https://issues.apache.org/jira/browse/HBASE-21515 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Duo Zhang Plan to do this incrementally, so first we will only introduce the new AsyncClusterConnection, without deleting the old ClusterConnection. And we can move the code which uses the ClusterConnection to use AsyncClusterConnection, through different sub tasks here. And finally we can remove the ClusterConnection completely. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699768#comment-16699768
]
Duo Zhang commented on HBASE-21513:
---
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
2018-06-18T02:33:14+08:00)
4.15.0-39-generic #42-Ubuntu SMP Tue Oct 23 15:48:01 UTC 2018 x86_64 x86_64
x86_64 GNU/Linux
Ubuntu 18.04.1 LTS \n \l
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Status: Patch Available (was: Open)
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.2, 1.2.7
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Status: Open (was: Patch Available)
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.2, 1.2.7
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Attachment: HBASE-21492.2.patch
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Attachment: (was: HBASE-21492.2.patch)
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Attachment: HBASE-21492.2.patch
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.2.7, 2.0.2
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Status: Open (was: Patch Available)
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.2, 1.2.7
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21492) CellCodec Written To WAL Before It's Verified
[
https://issues.apache.org/jira/browse/HBASE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HBASE-21492:
Status: Patch Available (was: Open)
> CellCodec Written To WAL Before It's Verified
> -
>
> Key: HBASE-21492
> URL: https://issues.apache.org/jira/browse/HBASE-21492
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.0.2, 1.2.7
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Critical
> Attachments: HBASE-21492.1.patch, HBASE-21492.2.patch
>
>
> The cell codec class name is written into the WAL file, but the cell codec
> class is not actually verified to exist. Therefore, users can inadvertently
> configure an invalid class name and it will be recorded into the WAL file.
> At that point, the WAL file becomes unreadable and blocks processing of all
> other WAL files.
> {code:java|title=AbstractProtobufLogWriter.java}
> private WALHeader buildWALHeader0(Configuration conf, WALHeader.Builder
> builder) {
> if (!builder.hasWriterClsName()) {
> builder.setWriterClsName(getWriterClassName());
> }
> if (!builder.hasCellCodecClsName()) {
> builder.setCellCodecClsName(WALCellCodec.getWALCellCodecClass(conf));
> }
> return builder.build();
> }
> {code}
> https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractProtobufLogWriter.java#L78-L86
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699746#comment-16699746
]
Sean Busbey commented on HBASE-21513:
-
what OS and version was the linux VM?
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699713#comment-16699713
]
stack commented on HBASE-21513:
---
bq. I'd really like the chance to figure out the underlying issue rather than
adding an entire additional build invocation just because the current symptom
goes away.
Be my guest sir.
Meantime, can use the patch attached here.
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699688#comment-16699688
]
Sean Busbey commented on HBASE-21513:
-
I'd really like the chance to figure out the underlying issue rather than
adding an entire additional build invocation just because the current symptom
goes away.
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[
https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699668#comment-16699668
]
Hudson commented on HBASE-21511:
Results for branch branch-1.3
[build #555 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/555/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/555//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/555//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/555//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
> ---
>
> Key: HBASE-21511
> URL: https://issues.apache.org/jira/browse/HBASE-21511
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.9, 2.1.2
>
> Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt,
> 21511.v1.txt, 21511.v2.txt, 21511.v3.txt
>
>
> During review of HBASE-21387, [~Apache9] mentioned that the check for in
> progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer
> needed now that snapshot hfile cleaner and taking snapshot are mutually
> exclusive.
> This issue is to address the review comment by removing the check for in
> progress snapshots.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699592#comment-16699592
]
Hadoop QA commented on HBASE-21513:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
54s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
0s{color} | {color:red} The patch generated 1 new + 9 unchanged - 0 fixed = 10
total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 4m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21513 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949549/make_rc.sh.txt |
| Optional Tests | dupname asflicense shellcheck shelldocs |
| uname | Linux ccb936118c7e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1acbd36c90 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| shellcheck | v0.4.4 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HBASE-Build/15120/artifact/patchprocess/diff-patch-shellcheck.txt
|
| Max. process+thread count | 48 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15120/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699583#comment-16699583
]
stack commented on HBASE-21513:
---
This patch worked for me (does the [~Apache9] suggestion).
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21513:
--
Attachment: make_rc.sh.txt
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21513:
--
Assignee: stack
Status: Patch Available (was: Open)
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: make_rc.sh.txt
>
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-18735) Provide a fast mechanism for shutting down mini cluster
[ https://issues.apache.org/jira/browse/HBASE-18735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Ervits reassigned HBASE-18735: Assignee: Artem Ervits > Provide a fast mechanism for shutting down mini cluster > --- > > Key: HBASE-18735 > URL: https://issues.apache.org/jira/browse/HBASE-18735 > Project: HBase > Issue Type: Wish >Reporter: Samarth Jain >Assignee: Artem Ervits >Priority: Major > > The current mechanism of shutting down a mini cluster through > HBaseTestingUtility.shutDownMiniCluster can take a lot of time when the mini > cluster almost has a lot of tables. A lot of this time is spent in closing > all the user regions. It would be nice to have a mechanism where this > shutdown can happen quickly without having to worry about closing these user > regions. At the same time, this mechanism would need to make sure that all > the critical system resources like file handles and network ports are still > released so that subsequently initialized mini clusters on the same JVM or > system won't run into resource issues. This would make testing using HBase > mini clusters much faster and immensely help out test frameworks of dependent > projects like Phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21501) Add “force-stop” to skip all close region logic in HBaseMiniCluster
[ https://issues.apache.org/jira/browse/HBASE-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artem Ervits resolved HBASE-21501. -- Resolution: Duplicate will track in the original Jira https://issues.apache.org/jira/browse/HBASE-18735 > Add “force-stop” to skip all close region logic in HBaseMiniCluster > --- > > Key: HBASE-21501 > URL: https://issues.apache.org/jira/browse/HBASE-21501 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Artem Ervits >Assignee: Artem Ervits >Priority: Major > > spoke to [~elserj] as a result of Phoenix dev meetup, this may be a much > needed feature. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699455#comment-16699455
]
Sean Busbey commented on HBASE-21513:
-
Could you either spin up the linux VM again or describe what you did so that I
can recreate it? this sounds very much like a maven version dependent thing.
[~Apache9] what version of ubuntu and where did you get maven from? (or just
what version of maven)
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Priority: Major
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699427#comment-16699427
]
stack commented on HBASE-21513:
---
Let me try your suggestion next time [~Apache9]
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Priority: Major
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[ https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699398#comment-16699398 ] Hudson commented on HBASE-21511: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #505 (See [https://builds.apache.org/job/HBase-1.3-IT/505/]) HBASE-21511 Remove in progress snapshot check in (yuzhihong: rev b4250dc2a98ded8ca9be6a3172977b99594e8313) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java > Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles > --- > > Key: HBASE-21511 > URL: https://issues.apache.org/jira/browse/HBASE-21511 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.9, 2.1.2 > > Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt, > 21511.v1.txt, 21511.v2.txt, 21511.v3.txt > > > During review of HBASE-21387, [~Apache9] mentioned that the check for in > progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer > needed now that snapshot hfile cleaner and taking snapshot are mutually > exclusive. > This issue is to address the review comment by removing the check for in > progress snapshots. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21513) [rm] make_rc.sh doesn't work on mac os x
[
https://issues.apache.org/jira/browse/HBASE-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699404#comment-16699404
]
stack commented on HBASE-21513:
---
I tried 3.6 and 3.5.4 on mac os x. Didn't register what linux version was.
> [rm] make_rc.sh doesn't work on mac os x
>
>
> Key: HBASE-21513
> URL: https://issues.apache.org/jira/browse/HBASE-21513
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: stack
>Priority: Major
>
> Trying to build an RC on a mac, it fails always for me with:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.6:single (default-cli) on
> project hbase-assembly: Failed to create assembly: Error adding file
> 'org.apache.hbase:hbase-common:jar:tests:2.0.3' to archive:
> /Users/stack/checkouts/hbase.git/hbase-common/target/test-classes isn't a
> file. -> [Help 1]
> [ERROR]
> {code}
> This is the second build that tries to assemble the tgz inside in make_rc.sh.
> If I leave out 'site' target, it works. I tried an earlier version that
> current head of branch-2.0 and it had same issue.
> [~busbey] had a nice suggestion changing the -DskipTests to
> -Dtest=NO_SUCH_TEST ... but that didn't work for me.
> I went and got a linux vm and it just worked.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[ https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21511: --- Fix Version/s: 1.4.9 1.3.3 > Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles > --- > > Key: HBASE-21511 > URL: https://issues.apache.org/jira/browse/HBASE-21511 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.9, 2.1.2 > > Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt, > 21511.v1.txt, 21511.v2.txt, 21511.v3.txt > > > During review of HBASE-21387, [~Apache9] mentioned that the check for in > progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer > needed now that snapshot hfile cleaner and taking snapshot are mutually > exclusive. > This issue is to address the review comment by removing the check for in > progress snapshots. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21505) Several inconsistencies on information reported for Replication Sources by hbase shell status 'replication' command.
[
https://issues.apache.org/jira/browse/HBASE-21505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699375#comment-16699375
]
Wellington Chevreuil commented on HBASE-21505:
--
Thanks for the comments [~tianjingyun], please refer to below:
{quote}How do you calculate replication lag when there is no entry need to
replicate for this peer? I saw you remove the code that update
TimeStampOfLastShippedOp{quote}
*TimeStampOfLastShippedOp* is still used for calculating replication lag, and
it's still updated on ReplicationSourceShipper only, once it has successfully
shipped the given edit. I had added this TimeStampOfLastAttempted to track the
timestamp of the latest entry added to source that should be replicated, so
that I can compare with shipment times and calculate the lag. As long as
shipment time is newer than time from latest added to source, there's no lag.
If arrival time is newer, then the lag is defined as: (current time - arrival
time);
{quote}getTimeStampOfLastAttemtped what is this for? I didn't see any place
that update this metric.{quote}
As explained before, it is being used to track timestamps of insertion, in
source, of last edit targeted for replication. I should rename this to
something more intuitive, such as TimeStampOfLastAddedToSource. It's updated on
ReplicationSourceWALReader.addEntryToBatch whenever new entry is read and
placed in the queue to be consumed by Shipper. It may be kept internally only,
though. Not sure if it's worth expose it on ;
{quote}same question for connectedToPeer{quote}
In the event where peer ZK is not available, ReplicationSource initialization
can get stuck in trying to connect to ZK, before it even starts trying to
read/ship edits, so thought about add this metric to give more insight on what
might be wrong with replication. Since workers had not really started, I
thought it wouldn't make much sense show 0 valued stats, but rather explicit
mention some initialization issues.
Yeah, patch attached was not yet intended for a commit. Still need to polish a
bit on variable names, what actually expose for hbase shell command and also
implement tests for the different conditions. Am planning to add an initial
patch version addressing that by tomorrow EOD. Let me know on any concerns you
might have about what we had discussed so far.
> Several inconsistencies on information reported for Replication Sources by
> hbase shell status 'replication' command.
>
>
> Key: HBASE-21505
> URL: https://issues.apache.org/jira/browse/HBASE-21505
> Project: HBase
> Issue Type: Bug
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Attachments:
> 0001-HBASE-21505-initial-version-for-more-detailed-report.patch
>
>
> While reviewing hbase shell status 'replication' command, noticed the
> following issues related to replication source section:
> 1) TimeStampsOfLastShippedOp keeps getting updated and increasing even when
> no new edits were added to source, so nothing was really shipped. Test steps
> performed:
> 1.1) Source cluster with only one table targeted to replication;
> 1.2) Added a new row, confirmed the row appeared in Target cluster;
> 1.3) Issued status 'replication' command in source, TimeStampsOfLastShippedOp
> shows current timestamp T1.
> 1.4) Waited 30 seconds, no new data added to source. Issued status
> 'replication' command, now shows timestamp T2.
> 2) When replication is stuck due some connectivity issues or target
> unavailability, if new edits are added in source, reported AgeOfLastShippedOp
> is wrongly showing same value as "Replication Lag". This is incorrect,
> AgeOfLastShippedOp should not change until there's indeed another edit
> shipped to target. Test steps performed:
> 2.1) Source cluster with only one table targeted to replication;
> 2.2) Stopped target cluster RS;
> 2.3) Put a new row on source. Running status 'replication' command does show
> lag increasing. TimeStampsOfLastShippedOp seems correct also, no further
> updates as described on bullet #1 above.
> 2.4) AgeOfLastShippedOp keeps increasing together with Replication Lag, even
> though there's no new edit shipped to target:
> {noformat}
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=5581, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=5581
> ...
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=8586, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=8586
> ...
> {noformat}
> 3) AgeOfLastShippedOp gets set to 0 even when a given edit had taken some
> time before it got finally shipped to target. Test steps performed:
> 3.1) Source cluster with
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[ https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699236#comment-16699236 ] Josh Elser commented on HBASE-21511: [~yuzhih...@gmail.com], is this unneeded on branch-1.2, branch-1.3, and branch-1.4? Confused why it didn't land on all of the same branches that HBASE-21387 did. > Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles > --- > > Key: HBASE-21511 > URL: https://issues.apache.org/jira/browse/HBASE-21511 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2 > > Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt, > 21511.v1.txt, 21511.v2.txt, 21511.v3.txt > > > During review of HBASE-21387, [~Apache9] mentioned that the check for in > progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer > needed now that snapshot hfile cleaner and taking snapshot are mutually > exclusive. > This issue is to address the review comment by removing the check for in > progress snapshots. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21387) Race condition surrounding in progress snapshot handling in snapshot cache leads to loss of snapshot files
[
https://issues.apache.org/jira/browse/HBASE-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699230#comment-16699230
]
Josh Elser commented on HBASE-21387:
Nevermind, I see HBASE-21511 was filed for this.
> Race condition surrounding in progress snapshot handling in snapshot cache
> leads to loss of snapshot files
> --
>
> Key: HBASE-21387
> URL: https://issues.apache.org/jira/browse/HBASE-21387
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Labels: snapshot
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.10
>
> Attachments: 0001-UT.patch, 21387-suggest.txt, 21387.addendum.txt,
> 21387.dbg.txt, 21387.v10.txt, 21387.v11.txt, 21387.v12.txt, 21387.v2.txt,
> 21387.v3.txt, 21387.v6.txt, 21387.v7.txt, 21387.v8.txt, 21387.v9.txt,
> 21511.v2.txt, HBASE-21387.branch-1.2.patch, HBASE-21387.branch-1.3.patch,
> HBASE-21387.branch-1.patch, HBASE-21387.v13.patch, HBASE-21387.v14.patch,
> HBASE-21387.v15.patch, HBASE-21387.v16.patch, HBASE-21387.v17.patch,
> two-pass-cleaner.v4.txt, two-pass-cleaner.v6.txt, two-pass-cleaner.v9.txt
>
>
> During recent report from customer where ExportSnapshot failed:
> {code}
> 2018-10-09 18:54:32,559 ERROR [VerifySnapshot-pool1-t2]
> snapshot.SnapshotReferenceUtil: Can't find hfile:
> 44f6c3c646e84de6a63fe30da4fcb3aa in the real
> (hdfs://in.com:8020/apps/hbase/data/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> or archive
> (hdfs://in.com:8020/apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> directory for the primary table.
> {code}
> We found the following in log:
> {code}
> 2018-10-09 18:54:23,675 DEBUG
> [00:16000.activeMasterManager-HFileCleaner.large-1539035367427]
> cleaner.HFileCleaner: Removing:
> hdfs:///apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa
> from archive
> {code}
> The root cause is race condition surrounding in progress snapshot(s) handling
> between refreshCache() and getUnreferencedFiles().
> There are two callers of refreshCache: one from RefreshCacheTask#run and the
> other from SnapshotHFileCleaner.
> Let's look at the code of refreshCache:
> {code}
> if (!name.equals(SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME)) {
> {code}
> whose intention is to exclude in progress snapshot(s).
> Suppose when the RefreshCacheTask runs refreshCache, there is some in
> progress snapshot (about to finish).
> When SnapshotHFileCleaner calls getUnreferencedFiles(), it sees that
> lastModifiedTime is up to date. So cleaner proceeds to check in progress
> snapshot(s). However, the snapshot has completed by that time, resulting in
> some file(s) deemed unreferenced.
> Here is timeline given by Josh illustrating the scenario:
> At time T0, we are checking if F1 is referenced. At time T1, there is a
> snapshot S1 in progress that is referencing a file F1. refreshCache() is
> called, but no completed snapshot references F1. At T2, the snapshot S1,
> which references F1, completes. At T3, we check in-progress snapshots and S1
> is not included. Thus, F1 is marked as unreferenced even though S1 references
> it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21387) Race condition surrounding in progress snapshot handling in snapshot cache leads to loss of snapshot files
[
https://issues.apache.org/jira/browse/HBASE-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699220#comment-16699220
]
Josh Elser commented on HBASE-21387:
bq. We will stop checking unreferenced files if there are snapshot operations
in progress, but in the code below we will get snapshots in progress and try to
filter out files...
Hrm, maybe I was too quick in my review. You're saying we're doing extra work
when there's a snapshot in progress when we could just short-circuit the entire
operation?
> Race condition surrounding in progress snapshot handling in snapshot cache
> leads to loss of snapshot files
> --
>
> Key: HBASE-21387
> URL: https://issues.apache.org/jira/browse/HBASE-21387
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Labels: snapshot
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.0.3, 1.4.9, 2.1.2, 1.2.10
>
> Attachments: 0001-UT.patch, 21387-suggest.txt, 21387.addendum.txt,
> 21387.dbg.txt, 21387.v10.txt, 21387.v11.txt, 21387.v12.txt, 21387.v2.txt,
> 21387.v3.txt, 21387.v6.txt, 21387.v7.txt, 21387.v8.txt, 21387.v9.txt,
> 21511.v2.txt, HBASE-21387.branch-1.2.patch, HBASE-21387.branch-1.3.patch,
> HBASE-21387.branch-1.patch, HBASE-21387.v13.patch, HBASE-21387.v14.patch,
> HBASE-21387.v15.patch, HBASE-21387.v16.patch, HBASE-21387.v17.patch,
> two-pass-cleaner.v4.txt, two-pass-cleaner.v6.txt, two-pass-cleaner.v9.txt
>
>
> During recent report from customer where ExportSnapshot failed:
> {code}
> 2018-10-09 18:54:32,559 ERROR [VerifySnapshot-pool1-t2]
> snapshot.SnapshotReferenceUtil: Can't find hfile:
> 44f6c3c646e84de6a63fe30da4fcb3aa in the real
> (hdfs://in.com:8020/apps/hbase/data/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> or archive
> (hdfs://in.com:8020/apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> directory for the primary table.
> {code}
> We found the following in log:
> {code}
> 2018-10-09 18:54:23,675 DEBUG
> [00:16000.activeMasterManager-HFileCleaner.large-1539035367427]
> cleaner.HFileCleaner: Removing:
> hdfs:///apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa
> from archive
> {code}
> The root cause is race condition surrounding in progress snapshot(s) handling
> between refreshCache() and getUnreferencedFiles().
> There are two callers of refreshCache: one from RefreshCacheTask#run and the
> other from SnapshotHFileCleaner.
> Let's look at the code of refreshCache:
> {code}
> if (!name.equals(SnapshotDescriptionUtils.SNAPSHOT_TMP_DIR_NAME)) {
> {code}
> whose intention is to exclude in progress snapshot(s).
> Suppose when the RefreshCacheTask runs refreshCache, there is some in
> progress snapshot (about to finish).
> When SnapshotHFileCleaner calls getUnreferencedFiles(), it sees that
> lastModifiedTime is up to date. So cleaner proceeds to check in progress
> snapshot(s). However, the snapshot has completed by that time, resulting in
> some file(s) deemed unreferenced.
> Here is timeline given by Josh illustrating the scenario:
> At time T0, we are checking if F1 is referenced. At time T1, there is a
> snapshot S1 in progress that is referencing a file F1. refreshCache() is
> called, but no completed snapshot references F1. At T2, the snapshot S1,
> which references F1, completes. At T3, we check in-progress snapshots and S1
> is not included. Thus, F1 is marked as unreferenced even though S1 references
> it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21487) Concurrent modify table ops can lead to unexpected results
[ https://issues.apache.org/jira/browse/HBASE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699180#comment-16699180 ] Allan Yang commented on HBASE-21487: I think we could record the seqenceID of the current tabledescriptor on FS(and pass to ModifyTableProcedure), then in prepare state of ModifyTableProcedure, we can check whether if the sequenceID of the current table descriptor from FS is different from the ID we recorded. If they are not equal, that means someone has already changed the descriptor, we can abort the modify procedure if so. > Concurrent modify table ops can lead to unexpected results > -- > > Key: HBASE-21487 > URL: https://issues.apache.org/jira/browse/HBASE-21487 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.0 >Reporter: Syeda Arshiya Tabreen >Priority: Major > > Concurrent modifyTable or add/delete/modify columnFamily leads to incorrect > result. After HBASE-18893, The behavior of add/delete/modify column family > during concurrent operation is changed compare to branch-1.When one client > is adding cf2 and another one cf3 .. In branch-1 final result will be > cf1,cf2,cf3 but now either cf1,cf2 OR cf1,cf3 will be the outcome depending > on which ModifyTableProcedure executed finally.Its because new table > descriptor is constructed before submitting the ModifyTableProcedure in > HMaster class and its not guarded by any lock. > *Steps to reproduce* > 1.Create table 't' with column family 'f1' > 2.Client-1 and Client-2 requests to add column family 'f2' and 'f3' on table > 't' concurrently. > *Expected Result* > Table should have three column families(f1,f2,f3) > *Actual Result* > Table 't' will have column family either (f1,f2) or (f1,f3) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699014#comment-16699014
]
Duo Zhang commented on HBASE-21154:
---
You can either leave comments here or on the review board, I will open follow
on issues to address them. The documentation is a good one.
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699010#comment-16699010
]
Sean Busbey commented on HBASE-21154:
-
bq. HBASE-21508 has been committed, so let me commit the patch here first.
Since it is only on master, I think it is OK as there is no recent releases so
we can open new issues to address the new problems.
Am I as a reviewer expected to open these new issues? Without looking at
anything other than the file list I can say that this change doesn't have
sufficient documentation. The namespace table going away is A Big Deal
operationally. It should be called out in an upgrade section on moving to 3.0.
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699005#comment-16699005
]
Sean Busbey commented on HBASE-21154:
-
bq. Sean Busbey Do you have any cycles to review the patch? I plan to integrate
this along with HBASE-21508, so I can start another round of ITBLL.
It was a holiday in my locale and I've been offline. Hence my request for time.
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21487) Concurrent modify table ops can lead to unexpected results
[ https://issues.apache.org/jira/browse/HBASE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698982#comment-16698982 ] Syeda Arshiya Tabreen commented on HBASE-21487: --- No, I'm not preparing any patch as of now,I was just thinking about the solutions and did not get any clear approach. when do we release the table lock if we grab the lock before constructing the new table descriptor? As per my understanding, when we release the lock before submitting the procedure, the issue wont be solved as the descriptor will override and it will lead to same unexpected results.If we release after submitting the procedure,the ModifyTableProcedure won't start until the table lock is released.Is this right? > Concurrent modify table ops can lead to unexpected results > -- > > Key: HBASE-21487 > URL: https://issues.apache.org/jira/browse/HBASE-21487 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.0 >Reporter: Syeda Arshiya Tabreen >Priority: Major > > Concurrent modifyTable or add/delete/modify columnFamily leads to incorrect > result. After HBASE-18893, The behavior of add/delete/modify column family > during concurrent operation is changed compare to branch-1.When one client > is adding cf2 and another one cf3 .. In branch-1 final result will be > cf1,cf2,cf3 but now either cf1,cf2 OR cf1,cf3 will be the outcome depending > on which ModifyTableProcedure executed finally.Its because new table > descriptor is constructed before submitting the ModifyTableProcedure in > HMaster class and its not guarded by any lock. > *Steps to reproduce* > 1.Create table 't' with column family 'f1' > 2.Client-1 and Client-2 requests to add column family 'f2' and 'f3' on table > 't' concurrently. > *Expected Result* > Table should have three column families(f1,f2,f3) > *Actual Result* > Table 't' will have column family either (f1,f2) or (f1,f3) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21481) [acl] Superuser's permissions should not be granted or revoked by any non-su global admin
[
https://issues.apache.org/jira/browse/HBASE-21481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698908#comment-16698908
]
Hadoop QA commented on HBASE-21481:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
44s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 17s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
34s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}126m 30s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.access.TestRpcAccessChecks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949473/HBASE-21481.master.005.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux e961d0fed262 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1acbd36c90 |
| maven | version: Apache Maven 3.5.4
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[
https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698821#comment-16698821
]
Hudson commented on HBASE-21511:
Results for branch master
[build #629 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/629/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
> ---
>
> Key: HBASE-21511
> URL: https://issues.apache.org/jira/browse/HBASE-21511
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2
>
> Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt,
> 21511.v1.txt, 21511.v2.txt, 21511.v3.txt
>
>
> During review of HBASE-21387, [~Apache9] mentioned that the check for in
> progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer
> needed now that snapshot hfile cleaner and taking snapshot are mutually
> exclusive.
> This issue is to address the review comment by removing the check for in
> progress snapshots.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21154) Remove hbase:namespace table; fold it into hbase:meta
[
https://issues.apache.org/jira/browse/HBASE-21154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698823#comment-16698823
]
Hudson commented on HBASE-21154:
Results for branch master
[build #629 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/629/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove hbase:namespace table; fold it into hbase:meta
> -
>
> Key: HBASE-21154
> URL: https://issues.apache.org/jira/browse/HBASE-21154
> Project: HBase
> Issue Type: Improvement
> Components: meta
>Reporter: stack
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21154-v1.patch, HBASE-21154-v2.patch,
> HBASE-21154-v4.patch, HBASE-21154-v5.patch, HBASE-21154-v6.patch,
> HBASE-21154-v7.patch, HBASE-21154.patch
>
>
> Namespace table is a small system table. Usually it has two rows. It must be
> assigned before user tables but after hbase:meta goes out. Its presence
> complicates our startup and is a constant source of grief when for whatever
> reason, it is not up and available. In fact, master startup is predicated on
> hbase:namespace being assigned and will not make progress unless it is up.
> Lets just add a new 'ns' column family to hbase:meta for namespace.
> Here is a default ns table content:
> {code}
> hbase(main):023:0* scan 'hbase:namespace'
> ROW
>COLUMN+CELL
> default
>column=info:d, timestamp=1526694059106,
> value=\x0A\x07default
> hbase
>column=info:d, timestamp=1526694059461,
> value=\x0A\x05hbase
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21498) Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a new BlockCache
[
https://issues.apache.org/jira/browse/HBASE-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698819#comment-16698819
]
Hadoop QA commented on HBASE-21498:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} hbase-server: The patch generated 0 new + 186
unchanged - 39 fixed = 186 total (was 225) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 30s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}287m 55s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}333m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.client.TestMobRestoreSnapshotFromClientAfterSplittingRegions |
| | hadoop.hbase.client.TestCloneSnapshotFromClientAfterSplittingRegion |
| | hadoop.hbase.master.procedure.TestServerCrashProcedureWithReplicas |
| | hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | hadoop.hbase.client.TestMobCloneSnapshotFromClientAfterSplittingRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21498 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949451/HBASE-21498.master.007.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 56c848ec0f9a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 27c0bf5c63 |
| maven | version: Apache Maven 3.5.4
[jira] [Commented] (HBASE-21505) Several inconsistencies on information reported for Replication Sources by hbase shell status 'replication' command.
[
https://issues.apache.org/jira/browse/HBASE-21505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698817#comment-16698817
]
Jingyun Tian commented on HBASE-21505:
--
[~wchevreuil] I just looke roughly through the whole patch and got some
questions.
# How do you calculate replication lag when there is no entry need to replicate
for this peer? I saw you remove the code that update TimeStampOfLastShippedOp
# *getTimeStampOfLastAttemtped* what is this for? I didn't see any place that
update this metric.
# same question for *connectedToPeer*
Looks like the patch is not finished yet. The core part of this is how to
calculate the replication lag correctly under different scenarios. Please take
this into consideration
> Several inconsistencies on information reported for Replication Sources by
> hbase shell status 'replication' command.
>
>
> Key: HBASE-21505
> URL: https://issues.apache.org/jira/browse/HBASE-21505
> Project: HBase
> Issue Type: Bug
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
> Attachments:
> 0001-HBASE-21505-initial-version-for-more-detailed-report.patch
>
>
> While reviewing hbase shell status 'replication' command, noticed the
> following issues related to replication source section:
> 1) TimeStampsOfLastShippedOp keeps getting updated and increasing even when
> no new edits were added to source, so nothing was really shipped. Test steps
> performed:
> 1.1) Source cluster with only one table targeted to replication;
> 1.2) Added a new row, confirmed the row appeared in Target cluster;
> 1.3) Issued status 'replication' command in source, TimeStampsOfLastShippedOp
> shows current timestamp T1.
> 1.4) Waited 30 seconds, no new data added to source. Issued status
> 'replication' command, now shows timestamp T2.
> 2) When replication is stuck due some connectivity issues or target
> unavailability, if new edits are added in source, reported AgeOfLastShippedOp
> is wrongly showing same value as "Replication Lag". This is incorrect,
> AgeOfLastShippedOp should not change until there's indeed another edit
> shipped to target. Test steps performed:
> 2.1) Source cluster with only one table targeted to replication;
> 2.2) Stopped target cluster RS;
> 2.3) Put a new row on source. Running status 'replication' command does show
> lag increasing. TimeStampsOfLastShippedOp seems correct also, no further
> updates as described on bullet #1 above.
> 2.4) AgeOfLastShippedOp keeps increasing together with Replication Lag, even
> though there's no new edit shipped to target:
> {noformat}
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=5581, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=5581
> ...
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=8586, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=8586
> ...
> {noformat}
> 3) AgeOfLastShippedOp gets set to 0 even when a given edit had taken some
> time before it got finally shipped to target. Test steps performed:
> 3.1) Source cluster with only one table targeted to replication;
> 3.2) Stopped target cluster RS;
> 3.3) Put a new row on source.
> 3.4) AgeOfLastShippedOp keeps increasing together with Replication Lag, even
> though there's no new edit shipped to target:
> {noformat}
> T1:
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=5581, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=5581
> ...
> T2:
> ...
> SOURCE: PeerID=1, AgeOfLastShippedOp=8586, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=8586
> ...
> {noformat}
> 3.5) Restart target cluster RS and verified the new row appeared there. No
> new edit added, but status 'replication' command reports AgeOfLastShippedOp
> as 0, while it should be the diff between the time it concluded shipping at
> target and the time it was added in source:
> {noformat}
> SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=1,
> TimeStampsOfLastShippedOp=Wed Nov 21 02:50:23 GMT 2018, Replication Lag=0
> {noformat}
> 4) When replication is stuck due some connectivity issues or target
> unavailability, if RS is restarted, once recovered queue source is started,
> TimeStampsOfLastShippedOp is set to initial java date (Thu Jan 01 01:00:00
> GMT 1970, for example), thus "Replication Lag" also gives a complete
> inaccurate value.
> Tests performed:
> 4.1) Source cluster with only one table targeted to replication;
> 4.2) Stopped target cluster RS;
> 4.3) Put a new row on source, restart RS on source, waited a few seconds for
> recovery queue source to startup, then it gives:
> {noformat}
> SOURCE:
[jira] [Commented] (HBASE-21508) Ignore the reportRegionStateTransition call from a dead server
[
https://issues.apache.org/jira/browse/HBASE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698822#comment-16698822
]
Hudson commented on HBASE-21508:
Results for branch master
[build #629 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/629/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/629//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Ignore the reportRegionStateTransition call from a dead server
> --
>
> Key: HBASE-21508
> URL: https://issues.apache.org/jira/browse/HBASE-21508
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21508-v1.patch, HBASE-21508-v2.patch,
> HBASE-21508-v3.patch, HBASE-21508-v3.patch, HBASE-21508-v4.patch,
> HBASE-21508.patch
>
>
> In our ITBLL test we observer a race between the SCP and TRSP which causes a
> region being assigned to two region servers.
> Not fully understand the scenario, but anyway, the we do not consider the
> situation in the old code, that after SCP gets the region list of a dead
> server, there could still be a reportRegionStateTransition call from dead
> server and mess up things.
> In general, I think we should have a fence in the AssignmentManager to
> prevent the reportRegionStateTransition from the dead servers to mess up the
> states.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[
https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698812#comment-16698812
]
Hudson commented on HBASE-21511:
Results for branch branch-2
[build #1524 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
> ---
>
> Key: HBASE-21511
> URL: https://issues.apache.org/jira/browse/HBASE-21511
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2
>
> Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt,
> 21511.v1.txt, 21511.v2.txt, 21511.v3.txt
>
>
> During review of HBASE-21387, [~Apache9] mentioned that the check for in
> progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer
> needed now that snapshot hfile cleaner and taking snapshot are mutually
> exclusive.
> This issue is to address the review comment by removing the check for in
> progress snapshots.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21508) Ignore the reportRegionStateTransition call from a dead server
[
https://issues.apache.org/jira/browse/HBASE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698813#comment-16698813
]
Hudson commented on HBASE-21508:
Results for branch branch-2
[build #1524 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1524//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Ignore the reportRegionStateTransition call from a dead server
> --
>
> Key: HBASE-21508
> URL: https://issues.apache.org/jira/browse/HBASE-21508
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21508-v1.patch, HBASE-21508-v2.patch,
> HBASE-21508-v3.patch, HBASE-21508-v3.patch, HBASE-21508-v4.patch,
> HBASE-21508.patch
>
>
> In our ITBLL test we observer a race between the SCP and TRSP which causes a
> region being assigned to two region servers.
> Not fully understand the scenario, but anyway, the we do not consider the
> situation in the old code, that after SCP gets the region list of a dead
> server, there could still be a reportRegionStateTransition call from dead
> server and mess up things.
> In general, I think we should have a fence in the AssignmentManager to
> prevent the reportRegionStateTransition from the dead servers to mess up the
> states.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21498) Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a new BlockCache
[ https://issues.apache.org/jira/browse/HBASE-21498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698809#comment-16698809 ] Anoop Sam John commented on HBASE-21498: Fine lets do the bigger change of CacheConfig refactor etc in a new issue. Lets fix the bigger issue fix . Can u pls raise the new one. > Master OOM when SplitTableRegionProcedure new CacheConfig and instantiate a > new BlockCache > -- > > Key: HBASE-21498 > URL: https://issues.apache.org/jira/browse/HBASE-21498 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21498.master.001.patch, > HBASE-21498.master.002.patch, HBASE-21498.master.003.patch, > HBASE-21498.master.004.patch, HBASE-21498.master.005.patch, > HBASE-21498.master.006.patch, HBASE-21498.master.006.patch, > HBASE-21498.master.007.patch, HBASE-21498.master.007.patch > > > In our cluster, we use a small heap/offheap config for master. After > HBASE-21290, master doesn't instantiate BlockCache when it not carry table. > But it will new CacheConfig in SplitTableRegionProcedure.splitStoreFiles > method. And it will instantiate a new BlockCache if it not initialized before > and make master OOM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[
https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698808#comment-16698808
]
Hudson commented on HBASE-21511:
Results for branch branch-2.1
[build #637 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/637/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/637//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/637//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/637//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
> ---
>
> Key: HBASE-21511
> URL: https://issues.apache.org/jira/browse/HBASE-21511
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2
>
> Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt,
> 21511.v1.txt, 21511.v2.txt, 21511.v3.txt
>
>
> During review of HBASE-21387, [~Apache9] mentioned that the check for in
> progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer
> needed now that snapshot hfile cleaner and taking snapshot are mutually
> exclusive.
> This issue is to address the review comment by removing the check for in
> progress snapshots.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21511) Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
[
https://issues.apache.org/jira/browse/HBASE-21511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698728#comment-16698728
]
Hudson commented on HBASE-21511:
Results for branch branch-1
[build #566 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/566/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/566//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/566//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/566//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Remove in progress snapshot check in SnapshotFileCache#getUnreferencedFiles
> ---
>
> Key: HBASE-21511
> URL: https://issues.apache.org/jira/browse/HBASE-21511
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.2
>
> Attachments: 21511.branch-1.v3.txt, 21511.branch-1.v4.txt,
> 21511.v1.txt, 21511.v2.txt, 21511.v3.txt
>
>
> During review of HBASE-21387, [~Apache9] mentioned that the check for in
> progress snapshots in SnapshotFileCache#getUnreferencedFiles is no longer
> needed now that snapshot hfile cleaner and taking snapshot are mutually
> exclusive.
> This issue is to address the review comment by removing the check for in
> progress snapshots.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21481) [acl] Superuser's permissions should not be granted or revoked by any non-su global admin
[
https://issues.apache.org/jira/browse/HBASE-21481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-21481:
--
Attachment: HBASE-21481.master.005.patch
> [acl] Superuser's permissions should not be granted or revoked by any non-su
> global admin
> -
>
> Key: HBASE-21481
> URL: https://issues.apache.org/jira/browse/HBASE-21481
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Labels: ACL, security-issue
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21481.master.001.patch,
> HBASE-21481.master.002.patch, HBASE-21481.master.003.patch,
> HBASE-21481.master.004.patch, HBASE-21481.master.005.patch
>
>
> Superusers are {{hbase.superuser}} listed in configuration and plus the one
> who start master process, these two may be overlap.
> A superuser must be a global admin, but a global admin may not be a
> superuser, possibly granted afterwards.
> For now, an non-su global admin with a Global.ADMIN permission can grant or
> revoke any superuser's permission, accidentally or deliberately.
> The purpose of this issue is to ban this action.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
