[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[
https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217453#comment-15217453
]
Vladimir Rodionov commented on HBASE-15509:
---
{quote}
Why have you selected a thread local? For a given writer the fsBlockWriter is
just one instance and when during flush/compaction we could use the same
fsBlockWriter only. So can we try to solve this without ThreadLocal?
{quote}
Let me think about this :)
> Avoid copy of block data in HFileBlock$Writer.finishBlock
> -
>
> Key: HBASE-15509
> URL: https://issues.apache.org/jira/browse/HBASE-15509
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-15509-v1.patch, HBASE-15509-v2.patch
>
>
> This calls ByteArrayOutputStream.toByteArray() which creates a copy of data
> block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[
https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217451#comment-15217451
]
Vladimir Rodionov commented on HBASE-15509:
---
{quote}
Any perf numbers? Any GC graphs? We do better with patch?
{quote}
We do better with patch. We reduced garbage allocation by 20-30% in compaction
and memstore flush threads. These are the only numbers I care about.
> Avoid copy of block data in HFileBlock$Writer.finishBlock
> -
>
> Key: HBASE-15509
> URL: https://issues.apache.org/jira/browse/HBASE-15509
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-15509-v1.patch, HBASE-15509-v2.patch
>
>
> This calls ByteArrayOutputStream.toByteArray() which creates a copy of data
> block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217435#comment-15217435 ] deepankar commented on HBASE-15525: --- Oh ok, sorry for the confusion thanks > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15554) StoreFile$Writer.appendGeneralBloomFilter generates extra KV
[ https://issues.apache.org/jira/browse/HBASE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217436#comment-15217436 ] ramkrishna.s.vasudevan commented on HBASE-15554: Thank you. > StoreFile$Writer.appendGeneralBloomFilter generates extra KV > > > Key: HBASE-15554 > URL: https://issues.apache.org/jira/browse/HBASE-15554 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Vladimir Rodionov >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > Accounts for 10% memory allocation when BloomFilterType is ROWCOL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15554) StoreFile$Writer.appendGeneralBloomFilter generates extra KV
[ https://issues.apache.org/jira/browse/HBASE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-15554: -- Assignee: ramkrishna.s.vasudevan (was: Vladimir Rodionov) > StoreFile$Writer.appendGeneralBloomFilter generates extra KV > > > Key: HBASE-15554 > URL: https://issues.apache.org/jira/browse/HBASE-15554 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Vladimir Rodionov >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > > Accounts for 10% memory allocation when BloomFilterType is ROWCOL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217431#comment-15217431
]
Hudson commented on HBASE-15295:
FAILURE: Integrated in HBase-Trunk_matrix #813 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/813/])
HBASE-15295 MutateTableAccess.multiMutate() does not get high priority (enis:
rev afdfd1bd9c938fa4b5c2aa9346e559167d550785)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217425#comment-15217425 ] Anoop Sam John commented on HBASE-15525: No I mean.. You pointed out issue when a Cell size is > size of a buf from the fixed sized buffer pool. So in this case (in any case) when the current buf is not capable of taking whole of the bytes, we will write whatever possible to cur buf. Get next buf and write the remaining all or possible as per its size and so on. > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217420#comment-15217420 ] ramkrishna.s.vasudevan commented on HBASE-15509: Some comments and questions Why have you selected a thread local? For a given writer the fsBlockWriter is just one instance and when during flush/compaction we could use the same fsBlockWriter only. So can we try to solve this without ThreadLocal? bq.This does an array copy (only when we can't reuse TLS buffer), I think this comment should be changed. Allocating and copying will happen only when we can't reuse else the copy is anyway happening in order to rewrite the contents. bq.// TODO Why is this TODO? > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-15509-v1.patch, HBASE-15509-v2.patch > > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217417#comment-15217417 ] deepankar commented on HBASE-15525: --- bq. Sure I will correct that.. One issue is that when we have curBuf with remaining size 100 and there is a need for 110 bytes for a cell. So we will not use remaining 100 bytes from this buf but go to next BB. Will try to solve this also.. That will auto solve the one u found Will that still has the same the just acquired buffer will stay empty so will the next buffer as the buffers we are using are of fixed size ? Side note also verified from sample internal production data, there were some outliers which could trigger this (thought our data was clean enough). > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217404#comment-15217404 ] Anoop Sam John commented on HBASE-15509: Any perf numbers? Any GC graphs? We do better with patch? Reused buffer set to a thread local which may have perf impacts? > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-15509-v1.patch, HBASE-15509-v2.patch > > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217392#comment-15217392 ] Anoop Sam John commented on HBASE-15525: Sure I will correct that.. One issue is that when we have curBuf with remaining size 100 and there is a need for 110 bytes for a cell. So we will not use remaining 100 bytes from this buf but go to next BB. Will try to solve this also.. That will auto solve the one u found > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15525) OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts
[ https://issues.apache.org/jira/browse/HBASE-15525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217378#comment-15217378 ] deepankar commented on HBASE-15525: --- A minor comment on the patch for the PoolAwareByteBufferOutputStream CheckSizeAndGrow is not taking into account the extra needed, this could lead to exception if we are copying a byte array which is bigger than the size of the byte buffer we get from pool (which is possible in case of KVs as there is no limit on sizes). we should either have a loop in case we are copying byte arrays / buffers (copyMethod) so that expansion could happen multiple times in case of Larger size copy. > OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts > -- > > Key: HBASE-15525 > URL: https://issues.apache.org/jira/browse/HBASE-15525 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: deepankar >Assignee: Anoop Sam John >Priority: Critical > Attachments: WIP.patch > > > After HBASE-13819 the system some times run out of direct memory whenever > there is some network congestion or some client side issues. > This was because of pending RPCs in the RPCServer$Connection.responseQueue > and since all the responses in this queue hold a buffer for cellblock from > BoundedByteBufferPool this could takeup a lot of memory if the > BoundedByteBufferPool's moving average settles down towards a higher value > See the discussion here > [HBASE-13819-comment|https://issues.apache.org/jira/browse/HBASE-13819?focusedCommentId=15207822=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15207822] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217374#comment-15217374 ] Lars Hofhansl edited comment on HBASE-15506 at 3/30/16 4:51 AM: My point is that that's not a problem as such. That's how Java is designed to work. It's only a problem when it is problem :) As in, it definitely slows things down, causes long GC pauses, etc. I don't have to tell you, but for just completeness of the discussion here: The principle costs to the garbage collector are (1) tracing all objects from the "root" objects and (2) collecting all unreachable objects. Obviously #1 is expensive when many objects need to be traced, and #2 is expensive when objects have to be moved (for example to reduce memory fragmentation) 64KB objects do not worry me, even if we have many GBs of them, it's just not many references to track. Further since they are all the same size, we won't fragment the heap in bad ways. Reusing objects (IMHO) is simply a very questionable technique. Especially when you have to reset the objects, which is more expensive then fitting a new object into a free slot of the same size. For what's it's worth, I have seen bad behaviour during heavy loading phases. I was always able to configure the GC accordingly, though. In any case, we should be able to create some single server test workload that exhibits problems if there are any. Those are good tests to have anyway, not just a way to appease me. was (Author: lhofhansl): My point is that that's a problem as such. That's how Java is designed to work. It's only a problem when it is problem :) As in, it definitely slows things down, causes long GC pauses, etc. I don't have to tell you, but for just completeness of the discussion here: The principle costs to the garbage collector are (1) tracing all objects from the "root" objects and (2) collecting all unreachable objects. Obviously #1 is expensive when many objects need to be traced, and #2 is expensive when objects have to be moved (for example to reduce memory fragmentation) 64KB objects do not worry me, even if we have many GBs of them, it's just not many references to track. Further since they are all the same size, we won't fragment the heap in bad ways. Reusing objects (IMHO) is simply a very questionable technique. Especially when you have to reset the objects, which is more expensive then fitting a new object into a free slot of the same size. For what's it's worth, I have seen bad behaviour during heavy loading phases. I was always able to configure the GC accordingly, though. In any case, we should be able to create some single server test workload that exhibits problems if there are any. Those are good tests to have anyway, not just a way to appease me. > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217374#comment-15217374 ] Lars Hofhansl commented on HBASE-15506: --- My point is that that's a problem as such. That's how Java is designed to work. It's only a problem when it is problem :) As in, it definitely slows things down, causes long GC pauses, etc. I don't have to tell you, but for just completeness of the discussion here: The principle costs to the garbage collector are (1) tracing all objects from the "root" objects and (2) collecting all unreachable objects. Obviously #1 is expensive when many objects need to be traced, and #2 is expensive when objects have to be moved (for example to reduce memory fragmentation) 64KB objects do not worry me, even if we have many GBs of them, it's just not many references to track. Further since they are all the same size, we won't fragment the heap in bad ways. Reusing objects (IMHO) is simply a very questionable technique. Especially when you have to reset the objects, which is more expensive then fitting a new object into a free slot of the same size. For what's it's worth, I have seen bad behaviour during heavy loading phases. I was always able to configure the GC accordingly, though. In any case, we should be able to create some single server test workload that exhibits problems if there are any. Those are good tests to have anyway, not just a way to appease me. > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15509: -- Attachment: HBASE-15509-v2.patch v2. Rebased to current master. > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-15509-v1.patch, HBASE-15509-v2.patch > > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15324) Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split
[
https://issues.apache.org/jira/browse/HBASE-15324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217345#comment-15217345
]
Yu Li commented on HBASE-15324:
---
Back on this... will wait for one more day and get this in if no objections.
> Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy
> and trigger unexpected split
> --
>
> Key: HBASE-15324
> URL: https://issues.apache.org/jira/browse/HBASE-15324
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.1.3
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-15324.patch, HBASE-15324_v2.patch,
> HBASE-15324_v3.patch
>
>
> We introduce jitter for region split decision in HBASE-13412, but the
> following line in {{ConstantSizeRegionSplitPolicy}} may cause long value
> overflow if MAX_FILESIZE is specified to Long.MAX_VALUE:
> {code}
> this.desiredMaxFileSize += (long)(desiredMaxFileSize * (RANDOM.nextFloat() -
> 0.5D) * jitter);
> {code}
> In our case we specify MAX_FILESIZE to Long.MAX_VALUE to prevent target
> region to split.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15529) Override needBalance in StochasticLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217337#comment-15217337
]
Hadoop QA commented on HBASE-15529:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 15s
{color} | {color:red} hbase-server: patch generated 3 new + 117 unchanged - 0
fixed = 120 total (was 117) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 56s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
| | hadoop.hbase.coprocessor.TestMasterObserver |
| | hadoop.hbase.TestRegionRebalancing |
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795848/HBASE-15529.patch |
| JIRA Issue | HBASE-15529 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217322#comment-15217322
]
Hudson commented on HBASE-15295:
FAILURE: Integrated in HBase-1.4 #61 (See
[https://builds.apache.org/job/HBase-1.4/61/])
HBASE-15295 MutateTableAccess.multiMutate() does not get high priority (enis:
rev 1e44dd82eb593d1d385ea52e7132c05f9405a5b1)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217316#comment-15217316
]
Hudson commented on HBASE-15295:
FAILURE: Integrated in HBase-1.3 #626 (See
[https://builds.apache.org/job/HBase-1.3/626/])
HBASE-15295 MutateTableAccess.multiMutate() does not get high priority (enis:
rev c0b9d00ee3d8e27fde39b20ed9c53c882e79)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>
[jira] [Commented] (HBASE-14920) Compacting Memstore
[
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217314#comment-15217314
]
Hadoop QA commented on HBASE-14920:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 212m 40s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 3m
5s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 292m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestPartialResultsFromClientSide |
| | hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort |
| | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217299#comment-15217299
]
Hudson commented on HBASE-15295:
FAILURE: Integrated in HBase-1.2 #591 (See
[https://builds.apache.org/job/HBase-1.2/591/])
HBASE-15295 MutateTableAccess.multiMutate() does not get high priority (enis:
rev 8d8a7107dc4ccbf36a92f64675dc60392f85c015)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
>
[jira] [Commented] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues
[
https://issues.apache.org/jira/browse/HBASE-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217292#comment-15217292
]
Hadoop QA commented on HBASE-15234:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 26s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
| | org.apache.hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795974/HBASE-15234.002.patch
|
| JIRA Issue | HBASE-15234 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux pietas.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / afdfd1b |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
[jira] [Commented] (HBASE-15481) Add pre/post roll to WALObserver
[
https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217283#comment-15217283
]
Hadoop QA commented on HBASE-15481:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 19s
{color} | {color:red} hbase-server: patch generated 2 new + 20 unchanged - 0
fixed = 22 total (was 20) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 6s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795970/HBASE-15481-v2.patch |
| JIRA Issue | HBASE-15481 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux pietas.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / afdfd1b |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
[jira] [Commented] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
[
https://issues.apache.org/jira/browse/HBASE-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217267#comment-15217267
]
Heng Chen commented on HBASE-15547:
---
{quote}
More generally I think the balancer should take into consideration a
regionserver's inability to serve regions.
{quote}
Agree. How to evaluate the RS inability? Yeah, "pending_open regions" is one
factor.
> Balancer should take into account number of PENDING_OPEN regions
>
>
> Key: HBASE-15547
> URL: https://issues.apache.org/jira/browse/HBASE-15547
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, Operability, Region Assignment
>Affects Versions: 0.98.0, 1.0.0
>Reporter: Sean Busbey
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
>
> We recently had a cluster get into a bad state where a subset of region
> servers consistently could not open new regions (but could continue serving
> the regions they already hosted).
> Recovering the cluster was just a matter of restarting region servers in
> sequence. However, this led to things getting substantially worse before they
> got better since the bulk assigner continued to place an uniform number of
> recovered regions across all servers, including onto those that could not
> open regions.
> It would be useful if the balancer could penalize regionservers with a
> backlog of pending_open regions and place more work on those regionservers
> that are properly serving.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[
https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217245#comment-15217245
]
Hadoop QA commented on HBASE-15509:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-15509 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795990/HBASE-15509-v1.patch |
| JIRA Issue | HBASE-15509 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1219/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Avoid copy of block data in HFileBlock$Writer.finishBlock
> -
>
> Key: HBASE-15509
> URL: https://issues.apache.org/jira/browse/HBASE-15509
> Project: HBase
> Issue Type: Sub-task
> Components: HFile
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-15509-v1.patch
>
>
> This calls ByteArrayOutputStream.toByteArray() which creates a copy of data
> block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues
[
https://issues.apache.org/jira/browse/HBASE-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217244#comment-15217244
]
Hadoop QA commented on HBASE-15234:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 134m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 190m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| | org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795955/HBASE-15234.001.patch
|
| JIRA Issue | HBASE-15234 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / afdfd1b |
|
[jira] [Commented] (HBASE-15265) Implement an asynchronous FSHLog
[
https://issues.apache.org/jira/browse/HBASE-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217228#comment-15217228
]
Duo Zhang commented on HBASE-15265:
---
{quote}
File issue to make this the default?
{quote}
I have already filed a issue, see HBASE-15536 :)
And mind taking a simple look at HBASE-15407 and HBASE-15538? We need to
support SASL and secure WAL otherwise AsyncFSWAL can not work under secure
environment thus can not be our default WAL I think? Thanks.
> Implement an asynchronous FSHLog
>
>
> Key: HBASE-15265
> URL: https://issues.apache.org/jira/browse/HBASE-15265
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15265-v1.patch, HBASE-15265-v2.patch,
> HBASE-15265-v3.patch, HBASE-15265-v4.patch, HBASE-15265-v5.patch,
> HBASE-15265-v6.patch, HBASE-15265-v7.patch, HBASE-15265-v8.patch,
> HBASE-15265-v8.patch, HBASE-15265-v8.patch, HBASE-15265-v8.patch,
> HBASE-15265-v8.patch, HBASE-15265-v8.patch, HBASE-15265-v8.patch,
> HBASE-15265-v8.patch, HBASE-15265.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217221#comment-15217221 ] Vladimir Rodionov commented on HBASE-15509: --- Good reduction in MemStore flusher threads > 35%. Compaction threads show less - 20%. Not bad anyway. > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-15509-v1.patch > > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15509: -- Attachment: HBASE-15509-v1.patch Patch v1. > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-15509-v1.patch > > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15509: -- Status: Patch Available (was: In Progress) > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-15509-v1.patch > > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15412) Add average region size metric
[ https://issues.apache.org/jira/browse/HBASE-15412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15412: --- Fix Version/s: 0.98.19 > Add average region size metric > -- > > Key: HBASE-15412 > URL: https://issues.apache.org/jira/browse/HBASE-15412 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Alicia Ying Shu > Fix For: 2.0.0, 1.3.0, 0.98.19, 1.4.0 > > Attachments: HBASE-15412.patch, HBASE-15412.patch, HBASE-15412.patch > > > We have several metrics related to region store file size, num regions, etc > per regionserver, but we do not have a single metric to track the average > region size per regionserver. > Avg region size is important to look at for deciding on the split policy, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15529) Override needBalance in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-15529: --- Status: Patch Available (was: Open) > Override needBalance in StochasticLoadBalancer > -- > > Key: HBASE-15529 > URL: https://issues.apache.org/jira/browse/HBASE-15529 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-15529.patch > > > StochasticLoadBalancer includes cost functions to compute the cost of region > rount, r/w qps, table load, region locality, memstore size, and storefile > size. Every cost function returns a number between 0 and 1 inclusive and the > computed costs are scaled by their respective multipliers. The bigger > multiplier means that the respective cost function have the bigger weight. > But needBalance decide whether to balance only by region count and doesn't > consider r/w qps, locality even you config these cost function with bigger > multiplier. StochasticLoadBalancer should override needBalance and decide > whether to balance by it's configs of cost functions. > Add one new config hbase.master.balancer.stochastic.minCostNeedBalance, > cluster need balance when (total cost / sum multiplier) > minCostNeedBalance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15529) Override needBalance in StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-15529: -- Assignee: Guanghao Zhang > Override needBalance in StochasticLoadBalancer > -- > > Key: HBASE-15529 > URL: https://issues.apache.org/jira/browse/HBASE-15529 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-15529.patch > > > StochasticLoadBalancer includes cost functions to compute the cost of region > rount, r/w qps, table load, region locality, memstore size, and storefile > size. Every cost function returns a number between 0 and 1 inclusive and the > computed costs are scaled by their respective multipliers. The bigger > multiplier means that the respective cost function have the bigger weight. > But needBalance decide whether to balance only by region count and doesn't > consider r/w qps, locality even you config these cost function with bigger > multiplier. StochasticLoadBalancer should override needBalance and decide > whether to balance by it's configs of cost functions. > Add one new config hbase.master.balancer.stochastic.minCostNeedBalance, > cluster need balance when (total cost / sum multiplier) > minCostNeedBalance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues
[
https://issues.apache.org/jira/browse/HBASE-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217193#comment-15217193
]
Andrew Purtell commented on HBASE-15234:
lgtm
"WarnOnlyAbortable" is a bit iffy
> ReplicationLogCleaner can abort due to transient ZK issues
> --
>
> Key: HBASE-15234
> URL: https://issues.apache.org/jira/browse/HBASE-15234
> Project: HBase
> Issue Type: Bug
> Components: master, Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Attachments: HBASE-15234.001.patch, HBASE-15234.002.patch
>
>
> The ReplicationLogCleaner delegate for the LogCleaner chore can abort due to
> transient errors reading the replication znodes, leaving the log cleaner
> chore stopped, but the master still running. This causes logs to build up in
> the oldWALs directory, which can even hit storage or file count limits in
> HDFS, causing problems.
> We've seen this happen in a couple of clusters when a rolling restart was
> performed on the zk peers (only one restarted at a time).
> The full stack trace when the log cleaner aborts is:
> {noformat}
> 16/02/02 15:22:39 WARN zookeeper.ZKUtil:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Unable to get data of znode /hbase/replication/rs
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/02/02 15:22:39 ERROR zookeeper.ZooKeeperWatcher:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
>
[jira] [Updated] (HBASE-15353) Add metric for number of CallQueueTooBigException's
[ https://issues.apache.org/jira/browse/HBASE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15353: --- Labels: beginner easy (was: easy noob) > Add metric for number of CallQueueTooBigException's > --- > > Key: HBASE-15353 > URL: https://issues.apache.org/jira/browse/HBASE-15353 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC, metrics >Affects Versions: 2.0.0, 1.2.0 >Reporter: Elliott Clark >Priority: Minor > Labels: beginner, easy > > This exception is being thrown more. We should add a metric for this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues
[
https://issues.apache.org/jira/browse/HBASE-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-15234:
--
Attachment: HBASE-15234.002.patch
Remove some extraneous imports that crept in.
> ReplicationLogCleaner can abort due to transient ZK issues
> --
>
> Key: HBASE-15234
> URL: https://issues.apache.org/jira/browse/HBASE-15234
> Project: HBase
> Issue Type: Bug
> Components: master, Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Attachments: HBASE-15234.001.patch, HBASE-15234.002.patch
>
>
> The ReplicationLogCleaner delegate for the LogCleaner chore can abort due to
> transient errors reading the replication znodes, leaving the log cleaner
> chore stopped, but the master still running. This causes logs to build up in
> the oldWALs directory, which can even hit storage or file count limits in
> HDFS, causing problems.
> We've seen this happen in a couple of clusters when a rolling restart was
> performed on the zk peers (only one restarted at a time).
> The full stack trace when the log cleaner aborts is:
> {noformat}
> 16/02/02 15:22:39 WARN zookeeper.ZKUtil:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Unable to get data of znode /hbase/replication/rs
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/02/02 15:22:39 ERROR zookeeper.ZooKeeperWatcher:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
>
[jira] [Commented] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
[ https://issues.apache.org/jira/browse/HBASE-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217097#comment-15217097 ] Elliott Clark commented on HBASE-15547: --- More generally I think the balancer should take into consideration a regionserver's inability to serve regions. We've seen issues where one regionserver was broken and just couldn't handle opening any region. So every time the balancer tried to move a region there it was causing un-availability. > Balancer should take into account number of PENDING_OPEN regions > > > Key: HBASE-15547 > URL: https://issues.apache.org/jira/browse/HBASE-15547 > Project: HBase > Issue Type: Improvement > Components: Balancer, Operability, Region Assignment >Affects Versions: 0.98.0, 1.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > > We recently had a cluster get into a bad state where a subset of region > servers consistently could not open new regions (but could continue serving > the regions they already hosted). > Recovering the cluster was just a matter of restarting region servers in > sequence. However, this led to things getting substantially worse before they > got better since the bulk assigner continued to place an uniform number of > recovered regions across all servers, including onto those that could not > open regions. > It would be useful if the balancer could penalize regionservers with a > backlog of pending_open regions and place more work on those regionservers > that are properly serving. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15481) Add pre/post roll to WALObserver
[ https://issues.apache.org/jira/browse/HBASE-15481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-15481: Attachment: HBASE-15481-v2.patch v2 is against current master where the FSHLog stuff moved to AbstractFSHLog > Add pre/post roll to WALObserver > > > Key: HBASE-15481 > URL: https://issues.apache.org/jira/browse/HBASE-15481 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.3.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15481-v0.patch, HBASE-15481-v1.patch, > HBASE-15481-v1.patch, HBASE-15481-v2.patch > > > currently the WALObserver has only a pre/post Write. It will be useful to > have a pre/post Roll too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217076#comment-15217076
]
Hudson commented on HBASE-15295:
SUCCESS: Integrated in HBase-1.2-IT #475 (See
[https://builds.apache.org/job/HBase-1.2-IT/475/])
HBASE-15295 MutateTableAccess.multiMutate() does not get high priority (enis:
rev 8d8a7107dc4ccbf36a92f64675dc60392f85c015)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
[jira] [Commented] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[ https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217053#comment-15217053 ] Zhan Zhang commented on HBASE-15333: [~jmhsieh] [~ted.m] Would you like to take a look? > Enhance the filter to handle short, integer, long, float and double > --- > > Key: HBASE-15333 > URL: https://issues.apache.org/jira/browse/HBASE-15333 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch, > HBASE-15333-3.patch, HBASE-15333-4.patch > > > Currently, the range filter is based on the order of bytes. But for java > primitive type, such as short, int, long, double, float, etc, their order is > not consistent with their byte order, extra manipulation has to be in place > to take care of them correctly. > For example, for the integer range (-100, 100), the filter <= 1, the current > filter will return 0 and 1, and the right return value should be (-100, 1] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217054#comment-15217054 ] Ben Manes commented on HBASE-15560: --- Hi Michael, I agree that offer a choice is an unnecessary confusion. It made it easier to test via a flag and, if retained, could be used only transitionally to give users a release cycle to adjust. For the 100% case LruBlockCache's work is a hash table read and increment of a global counter (fetch-and-add instruction). Caffeine's is the hash table read, select a ring buffer, CAS append into it, and schedule an async drain if full on ForkJoinPool#commonPool() (which might cause a few more context switches). The differences were in a margin of error in LruBlockCache's favor, but a tiny penalty seems understandable. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Reporter: Ben Manes > Attachments: tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217021#comment-15217021
]
Hudson commented on HBASE-15295:
SUCCESS: Integrated in HBase-1.3-IT #588 (See
[https://builds.apache.org/job/HBase-1.3-IT/588/])
HBASE-15295 MutateTableAccess.multiMutate() does not get high priority (enis:
rev 1e44dd82eb593d1d385ea52e7132c05f9405a5b1)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217028#comment-15217028 ] stack commented on HBASE-15560: --- Patch looks good. Not too interested in giving folks a choice for L1. Would rather just switch over if can demonstrate it generally better running Caffeine; theory-wise it would make sense. Why you think LRUBlockCache did better in 100% case [~ben.manes]? Caffeine is doing more work? If the two implementations are close when all comes from cache and Caffeine does better when there are misses, I'd say it no brainer. Let me do my own compare... Anyone else want to chime in here? [~mantonov]? > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Reporter: Ben Manes > Attachments: tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[
https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217022#comment-15217022
]
Hadoop QA commented on HBASE-15333:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 31s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 3s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795933/HBASE-15333-4.patch |
| JIRA Issue | HBASE-15333 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / afdfd1b |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/1213/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1213/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance the filter to handle short, integer, long, float and double
> ---
>
> Key: HBASE-15333
> URL: https://issues.apache.org/jira/browse/HBASE-15333
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zhan Zhang
>Assignee: Zhan Zhang
> Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch,
> HBASE-15333-3.patch, HBASE-15333-4.patch
>
>
> Currently, the range filter is based on the order of bytes. But for java
> primitive type, such as short, int, long, double, float, etc, their order is
> not consistent with their byte order, extra manipulation has to be in place
> to take care of them correctly.
> For example, for the integer range (-100, 100), the filter <= 1, the current
> filter will return 0 and 1, and the right return value should be (-100, 1]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217009#comment-15217009
]
Appy commented on HBASE-15236:
--
Ignore the flaky tests.
Ready for review ([Review board|https://reviews.apache.org/r/43873/]). Shower
your comments and/or +1s.
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236-master-v4.patch,
> HBASE-15236-master-v5.patch, HBASE-15236-master-v6.patch,
> HBASE-15236-master-v7.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues
[
https://issues.apache.org/jira/browse/HBASE-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-15234:
--
Status: Patch Available (was: Open)
> ReplicationLogCleaner can abort due to transient ZK issues
> --
>
> Key: HBASE-15234
> URL: https://issues.apache.org/jira/browse/HBASE-15234
> Project: HBase
> Issue Type: Bug
> Components: master, Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Attachments: HBASE-15234.001.patch
>
>
> The ReplicationLogCleaner delegate for the LogCleaner chore can abort due to
> transient errors reading the replication znodes, leaving the log cleaner
> chore stopped, but the master still running. This causes logs to build up in
> the oldWALs directory, which can even hit storage or file count limits in
> HDFS, causing problems.
> We've seen this happen in a couple of clusters when a rolling restart was
> performed on the zk peers (only one restarted at a time).
> The full stack trace when the log cleaner aborts is:
> {noformat}
> 16/02/02 15:22:39 WARN zookeeper.ZKUtil:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Unable to get data of znode /hbase/replication/rs
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/02/02 15:22:39 ERROR zookeeper.ZooKeeperWatcher:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
>
[jira] [Updated] (HBASE-15234) ReplicationLogCleaner can abort due to transient ZK issues
[
https://issues.apache.org/jira/browse/HBASE-15234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-15234:
--
Attachment: HBASE-15234.001.patch
Attaching a patch against master. This changes ReplicationLogCleaner to use a
"dummy" abortable for zookeeper operations, which will warn on errors, but
allow ReplicationLogCleaner to otherwise continue.
> ReplicationLogCleaner can abort due to transient ZK issues
> --
>
> Key: HBASE-15234
> URL: https://issues.apache.org/jira/browse/HBASE-15234
> Project: HBase
> Issue Type: Bug
> Components: master, Replication
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Critical
> Attachments: HBASE-15234.001.patch
>
>
> The ReplicationLogCleaner delegate for the LogCleaner chore can abort due to
> transient errors reading the replication znodes, leaving the log cleaner
> chore stopped, but the master still running. This causes logs to build up in
> the oldWALs directory, which can even hit storage or file count limits in
> HDFS, causing problems.
> We've seen this happen in a couple of clusters when a rolling restart was
> performed on the zk peers (only one restarted at a time).
> The full stack trace when the log cleaner aborts is:
> {noformat}
> 16/02/02 15:22:39 WARN zookeeper.ZKUtil:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Unable to get data of znode /hbase/replication/rs
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.getDeletableFiles(ReplicationLogCleaner.java:70)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
> at
> org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:110)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/02/02 15:22:39 ERROR zookeeper.ZooKeeperWatcher:
> replicationLogCleaner-0x1522c8b93c2fbae, quorum=,
> baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
> = ConnectionLoss for /hbase/replication/rs
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
> at
> org.apache.hadoop.hbase.replication.ReplicationQueuesClientZKImpl.getQueuesZNodeCversion(ReplicationQueuesClientZKImpl.java:80)
>
> at
> org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner.loadWALsFromQueues(ReplicationLogCleaner.java:99)
>
> at
>
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216953#comment-15216953
]
Heng Chen commented on HBASE-15406:
---
OK, i will do it.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14920) Compacting Memstore
[ https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14920: -- Attachment: move.to.junit4.patch Patch to integrate which changes TestMemStoreDefault to be junit4... if it helps. > Compacting Memstore > --- > > Key: HBASE-14920 > URL: https://issues.apache.org/jira/browse/HBASE-14920 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch, > move.to.junit4.patch > > > Implementation of a new compacting memstore with non-optimized immutable > segment representation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15191) CopyTable and VerifyReplication - Option to specify batch size, versions
[ https://issues.apache.org/jira/browse/HBASE-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216950#comment-15216950 ] Hudson commented on HBASE-15191: FAILURE: Integrated in HBase-1.4 #60 (See [https://builds.apache.org/job/HBase-1.4/60/]) HBASE-15191 CopyTable and VerifyReplication - Option to specify batch (tedyu: rev c7ad42867dbf35c5d07e4e3085edc2fd2145) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java > CopyTable and VerifyReplication - Option to specify batch size, versions > > > Key: HBASE-15191 > URL: https://issues.apache.org/jira/browse/HBASE-15191 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 0.98.16.1 >Reporter: Parth Shah >Assignee: Parth Shah >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE_15191.patch, HBASE_15191.patch > > > Need option to specify batch size for CopyTable and VerifyReplication. We > are working on patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216949#comment-15216949
]
Ted Yu commented on HBASE-15406:
Should be fine for now.
Once you get +1, it would be better to test on cluster before committing.
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216946#comment-15216946
]
Heng Chen commented on HBASE-15406:
---
I test it on standalone mode, is it enough? [~tedyu]
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-15295.
---
Resolution: Fixed
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> And the index region is trying to split, and is trying to do a meta update:
> {code}
> "regionserver//10.132.70.191:16020-splits-1454693389669" #1779
> prio=5 os_prio=0 tid=0x7f29e149c000 nid=0x5107 in Object.wait()
> [0x7f1f136d6000]
>
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216941#comment-15216941
]
Ted Yu commented on HBASE-15406:
Heng, thanks for your patience.
Can you try out patch v2 on cluster and post back the result ?
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
[ https://issues.apache.org/jira/browse/HBASE-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216932#comment-15216932 ] Sean Busbey commented on HBASE-15547: - it needn't even be that a RS can't open regions at all. just that some RS might severely lag others for a variety of reasons. If one of our goals is to keep MTTR reasonable, then a RS that is anomalous in how long it takes to get things open should be getting less work. > Balancer should take into account number of PENDING_OPEN regions > > > Key: HBASE-15547 > URL: https://issues.apache.org/jira/browse/HBASE-15547 > Project: HBase > Issue Type: Improvement > Components: Balancer, Operability, Region Assignment >Affects Versions: 0.98.0, 1.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > > We recently had a cluster get into a bad state where a subset of region > servers consistently could not open new regions (but could continue serving > the regions they already hosted). > Recovering the cluster was just a matter of restarting region servers in > sequence. However, this led to things getting substantially worse before they > got better since the bulk assigner continued to place an uniform number of > recovered regions across all servers, including onto those that could not > open regions. > It would be useful if the balancer could penalize regionservers with a > backlog of pending_open regions and place more work on those regionservers > that are properly serving. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15547) Balancer should take into account number of PENDING_OPEN regions
[ https://issues.apache.org/jira/browse/HBASE-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216926#comment-15216926 ] Heng Chen commented on HBASE-15547: --- Should balance policy take a bug (at least "RS not open regions" looks like a bug ) into account? > Balancer should take into account number of PENDING_OPEN regions > > > Key: HBASE-15547 > URL: https://issues.apache.org/jira/browse/HBASE-15547 > Project: HBase > Issue Type: Improvement > Components: Balancer, Operability, Region Assignment >Affects Versions: 0.98.0, 1.0.0 >Reporter: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.4.0 > > > We recently had a cluster get into a bad state where a subset of region > servers consistently could not open new regions (but could continue serving > the regions they already hosted). > Recovering the cluster was just a matter of restarting region servers in > sequence. However, this led to things getting substantially worse before they > got better since the bulk assigner continued to place an uniform number of > recovered regions across all servers, including onto those that could not > open regions. > It would be useful if the balancer could penalize regionservers with a > backlog of pending_open regions and place more work on those regionservers > that are properly serving. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15295:
--
Attachment: hbase-15295_addendum.patch
Test only fails in branch-1 and below, not in master it seems.
Will push this one-liner addendum shortly. Thanks [~saint@gmail.com] for
noticing.
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_addendum.patch, hbase-15295_v1.patch,
> hbase-15295_v1.patch, hbase-15295_v2.patch, hbase-15295_v3.patch,
> hbase-15295_v4.patch, hbase-15295_v5.patch, hbase-15295_v5.patch,
> hbase-15295_v6.patch, hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> And the index region is trying to split, and is trying to do a meta update:
>
[jira] [Updated] (HBASE-15333) Enhance the filter to handle short, integer, long, float and double
[ https://issues.apache.org/jira/browse/HBASE-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhan Zhang updated HBASE-15333: --- Attachment: HBASE-15333-4.patch > Enhance the filter to handle short, integer, long, float and double > --- > > Key: HBASE-15333 > URL: https://issues.apache.org/jira/browse/HBASE-15333 > Project: HBase > Issue Type: Sub-task >Reporter: Zhan Zhang >Assignee: Zhan Zhang > Attachments: HBASE-15333-1.patch, HBASE-15333-2.patch, > HBASE-15333-3.patch, HBASE-15333-4.patch > > > Currently, the range filter is based on the order of bytes. But for java > primitive type, such as short, int, long, double, float, etc, their order is > not consistent with their byte order, extra manipulation has to be in place > to take care of them correctly. > For example, for the integer range (-100, 100), the filter <= 1, the current > filter will return 0 and 1, and the right return value should be (-100, 1] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216908#comment-15216908
]
Heng Chen commented on HBASE-15406:
---
Relate test case has passed. And javac warnings have no relates with patch_V2.
Any concerns about patch_v2? [~tedyu] [~mantonov]
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14983) Create metrics for per block type hit/miss ratios
[
https://issues.apache.org/jira/browse/HBASE-14983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216898#comment-15216898
]
Hadoop QA commented on HBASE-14983:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 57s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 13s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 48s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
29s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 59s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 53s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} |
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216887#comment-15216887
]
Hadoop QA commented on HBASE-15236:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
34m 56s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 145m 10s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 215m 26s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | org.apache.hadoop.hbase.wal.TestDefaultWALProvider |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795883/HBASE-15236-master-v7.patch
|
| JIRA Issue | HBASE-15236 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf910.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7f39baf |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |
[jira] [Updated] (HBASE-15480) Bloom Filter check needs to be more efficient for array
[ https://issues.apache.org/jira/browse/HBASE-15480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Koetke updated HBASE-15480: -- Status: In Progress (was: Patch Available) > Bloom Filter check needs to be more efficient for array > --- > > Key: HBASE-15480 > URL: https://issues.apache.org/jira/browse/HBASE-15480 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 1.0.4 > > Attachments: HBASE-15480-branch-1.0.patch > > > It is currently inefficient to do lots of bloom filter checks. Each check has > overhead like going to the block cache to retrieve the block and recording > metrics. It would be good to have one bloom filter check api that does a > bunch of checks without so much block retrieval and metrics updates. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14920) Compacting Memstore
[
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216864#comment-15216864
]
stack commented on HBASE-14920:
---
s/startCompact/startCompaction/g
s/doCompact/doCompaction/g
No need of these things...
445
/*--
no one else does this...
Oh... so we could let go of an edit because it was beyond max versions or
something and then you need to update the lowest outstanding sequenceid
this thing?
465 updateLowestUnflushedSequenceIdInWal(true); // only if greater
I'd never have thought to do that.
We are changing the flush policy on the user here?
944 if(hasCompactingStore) {
945 double alpha = FLUSH_SIZE_WEIGHT;
946 this.memstoreFlushSize =
947 (long)(alpha*memstoreFlushSize +
(1-alpha)*blockingMemStoreSize);
948 LOG.info("Increasing memstore flush size to
"+memstoreFlushSize +" for the region="
949 + this);
950
htableDescriptor.setFlushPolicyClassName(FlushNonCompactingStoresFirstPolicy.class
951 .getName());
952 }
Thats not bad in-itself (just Release Note it as well as the fact that we are
going to run w/ more memory), but we are doing it by setting policy on the
HTableDescriptor; every region is going to do this? Can we do it on the region
itself? Or can we get away w/o changing HTD? This is unusual practice.
Do you have to start your own? Can you not piggyback off the RS's existing
Executor?
1748
this.service.startExecutorService(ExecutorType.RS_IN_MEMORY_FLUSH_AND_COMPACTION,
1749
conf.getInt("hbase.regionserver.executor.inmemoryflush.threads", 3));
Do you need to add this one to RegionServicesFor...
56public HTableDescriptor getTableDesc() {
57 return region.getTableDesc();
58}
Would be good if you could get away w/ not passing it down...
And this one...
68public RegionServerServices getRegionServerServices() {
69 return region.getRegionServerServices();
70}
This onlyIfGreater was there before you?
168 void updateStore(byte[] encodedRegionName, byte[] familyName, Long
sequenceId,
169 boolean onlyIfGreater) {
We should purge it? It is of little use?
TestCompactingMemStore is junit3 rather than junit4 because it parent is. I'll
attach a patch here that moves TestDefaultMemStore to junit4 so this test can
be junit 4 too (We've been trying to purge the old tests...)
I skimmed the tests. They look great.
Looks committable to me.
> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable
> segment representation
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15559) BaseMasterAndRegionObserver doesn't implement all the methods
[
https://issues.apache.org/jira/browse/HBASE-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216849#comment-15216849
]
Hadoop QA commented on HBASE-15559:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 6s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 37s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795892/HBASE-15559-v2.patch |
| JIRA Issue | HBASE-15559 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux pietas.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7f39baf |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 |
[jira] [Updated] (HBASE-15556) need extensible ConsistencyControl interface
[ https://issues.apache.org/jira/browse/HBASE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Koetke updated HBASE-15556: -- Status: In Progress (was: Patch Available) > need extensible ConsistencyControl interface > > > Key: HBASE-15556 > URL: https://issues.apache.org/jira/browse/HBASE-15556 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 1.0.4 > > Attachments: HBASE-15556-branch-1.0.patch > > > The class MultiVersionConsistencyControl should be abstracted into an > interface ConsistencyControl so it can be extended by a configured custom > implementation class, with MultiVersionConsistencyControl as the default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14920) Compacting Memstore
[ https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216812#comment-15216812 ] stack commented on HBASE-14920: --- I see/remember why the isCompactingMemStore method. Makes sense. Make it more generic? isSloppy or isRunsToFat or isNeedsHeadRoom. Minor. Can ignore. We can change when a new type shows up that is not compacting but that wants to also run close-to-the-edge... > Compacting Memstore > --- > > Key: HBASE-14920 > URL: https://issues.apache.org/jira/browse/HBASE-14920 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch > > > Implementation of a new compacting memstore with non-optimized immutable > segment representation -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216796#comment-15216796
]
Enis Soztutar commented on HBASE-15295:
---
I've ran it 20 times and still succeeds. I think you are running on 1.2. Let me
try that as well.
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch, hbase-15295_v6.patch,
> hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> And the index region is trying to split, and is trying to do a meta update:
> {code}
> "regionserver//10.132.70.191:16020-splits-1454693389669" #1779
> prio=5
[jira] [Commented] (HBASE-15498) Add conditions in MultiRowMutationService
[
https://issues.apache.org/jira/browse/HBASE-15498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216766#comment-15216766
]
Hadoop QA commented on HBASE-15498:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 20s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 33m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 18m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 6s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 43s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 8s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 49m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
0s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
103m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 4m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 19m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 14s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-15559) BaseMasterAndRegionObserver doesn't implement all the methods
[
https://issues.apache.org/jira/browse/HBASE-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216745#comment-15216745
]
Hadoop QA commented on HBASE-15559:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 31s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 159m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | org.apache.hadoop.hbase.security.access.TestNamespaceCommands |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795876/HBASE-15559-v1.patch |
| JIRA Issue | HBASE-15559 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15411) Rewrite backup with Procedure V2
[ https://issues.apache.org/jira/browse/HBASE-15411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216742#comment-15216742 ] Ted Yu commented on HBASE-15411: [~mbertozzi] [~enis]: Any more review comment ? Thanks > Rewrite backup with Procedure V2 > > > Key: HBASE-15411 > URL: https://issues.apache.org/jira/browse/HBASE-15411 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 15411-v1.txt, 15411-v11.txt, 15411-v12.txt, > 15411-v13.txt, 15411-v14.txt, 15411-v15.txt, 15411-v16.txt, 15411-v18.txt, > 15411-v22.txt, 15411-v3.txt, 15411-v5.txt, 15411-v6.txt, 15411-v7.txt, > 15411-v9.txt, FullTableBackupProcedure.java > > > Currently full / incremental backup is driven by BackupHandler (see call() > method for flow). > This issue is to rewrite the flow using Procedure V2. > States (enum) for full / incremental backup would be introduced in > Backup.proto which correspond to the steps performed in BackupHandler#call(). > executeFromState() would pace the backup based on the current state. > serializeStateData() / deserializeStateData() would be used to persist state > into procedure WAL. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216727#comment-15216727 ] Vladimir Rodionov commented on HBASE-15509: --- ... need to check independently data, index, bloom blocks. > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14920) Compacting Memstore
[
https://issues.apache.org/jira/browse/HBASE-14920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216720#comment-15216720
]
stack commented on HBASE-14920:
---
I like the notion of a 'CompactionMemStore' instead of a plain old MemStore.
Reading the nice description made me think of how we need to work on recovery.
There'll be more to recover if we can carry more in memory. Another issue.
This needs a comment on it: IN_MEMORY_FLUSH_THRESHOLD_FACTOR
Would be good if could use the Interface Store rather than implementation
HStore in below
66private HStore store;
For these...
// A flag for tests only
... we normally just flag with @VisibleForTesting annotation. FYI.
I've asked this before but forgot the answer, this has to be here:
isCompactingMemStore ?
Is this a race in the below?
184 @Override public long getFlushableSize() {
185 long snapshotSize = getSnapshot().getSize();
186 if(snapshotSize == 0) {
187 //if snapshot is empty the tail of the pipeline is flushed
188 snapshotSize = pipeline.getTailSize();
189 }
190 return snapshotSize > 0 ? snapshotSize : keySize();
191 }
If no snapshot, use the tail of the pipeline? The pipeline tail has not yet
been moved to be the new snapshot?
If sequenceid is MAX, don't we want to update it?
if(minSequenceId != Long.MAX_VALUE) {
I asked this before too...so, to do compaction, we need to have exclusive lock
or exclusive update lock just while you swap segments (the compacted for the
inputs?)... it looks like the next method, flushInMemory, answers my question
so ignore..
245 /* The thread is dispatched to flush-in-memory. This cannot be
done
246 * on the same thread, because for flush-in-memory we require
updatesLock
247 * in exclusive mode while this method (checkActiveSize) is
invoked holding updatesLock
248 * in the shared mode. */
Any guard against queuing same old compactions over and over again?
250 if(pool != null) { // the pool can be null in some tests
scenarios
251 InMemoryFlushWorker worker = new InMemoryFlushWorker();
252 LOG.info("Dispatching the MemStore in-memory flush for store "
253 + store.getColumnFamilyName());
254 pool.submit(worker);
255 inMemoryFlushInProgress.set(true);
256 }
... say there is a bug or we are slow to run already-queued compactions?
s/stopCompact/stopCompact/g
The below would seem to say we compact with updates lock held. Fix if wrong...
if it is true, then lets discuss:
351 * It takes the updatesLock exclusively, pushes active into the
pipeline,
352 * and compacts the pipeline.
Got as far as MemstoreCompactor... will be back.
> Compacting Memstore
> ---
>
> Key: HBASE-14920
> URL: https://issues.apache.org/jira/browse/HBASE-14920
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14920-V01.patch, HBASE-14920-V02.patch
>
>
> Implementation of a new compacting memstore with non-optimized immutable
> segment representation
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15509: -- Issue Type: Sub-task (was: Improvement) Parent: HBASE-1 > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Sub-task > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
[ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216710#comment-15216710 ] Vladimir Rodionov commented on HBASE-15509: --- Yes, that will only work if there is no cache-on-write. > Avoid copy of block data in HFileBlock$Writer.finishBlock > - > > Key: HBASE-15509 > URL: https://issues.apache.org/jira/browse/HBASE-15509 > Project: HBase > Issue Type: Improvement > Components: HFile >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > This calls ByteArrayOutputStream.toByteArray() which creates a copy of data > block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216708#comment-15216708
]
Hadoop QA commented on HBASE-15406:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 1s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 2s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 43s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 29m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 2s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 44s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 44s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 11s
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 4
new + 8 unchanged - 4 fixed = 12 total (was 12) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 11s
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 4
new + 8 unchanged - 4 fixed = 12 total (was 12) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 10m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216684#comment-15216684
]
stack commented on HBASE-15295:
---
Understood [~enis] Tests passed when you tried the patch but it fails for me
reliably:
{code}
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.master.TestMasterNoCluster
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 23.36 sec <<<
FAILURE! - in org.apache.hadoop.hbase.master.TestMasterNoCluster
testFailover(org.apache.hadoop.hbase.master.TestMasterNoCluster) Time elapsed:
22.3 sec <<< ERROR!
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /hbase/region-in-transition/1588230740
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
at
org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataNoWatch(ZKUtil.java:713)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:814)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:795)
at
org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeClosed(ZKAssign.java:566)
at
org.apache.hadoop.hbase.master.TestMasterNoCluster.testFailover(TestMasterNoCluster.java:244)
Results :
Tests in error:
TestMasterNoCluster.testFailover:244 » SessionExpired KeeperErrorCode =
Sessio...
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{code}
kalashnikov:hbase.git.commit stack$ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
On an old mac osx.
At this point:
commit 547095ab7325cab84f41f18dd64d22c2922e4727
Author: Sean Busbey
Date: Mon Mar 28 21:54:05 2016 -0500
HBASE-15500 update version to 1.2.1 for RC.
If I back out to the change after yours, still fails:
Author: chenheng
Date: Thu Mar 24 15:19:02 2016 +0800
HBASE-15384 Avoid using '/tmp' directory in TestBulkLoad
If I back out to just before this commit, it passes:
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.master.TestMasterNoCluster
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.617 sec - in
org.apache.hadoop.hbase.master.TestMasterNoCluster
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
On linux box, at head of 1.2, I get this when I run the test:
Tests in error:
TestMasterNoCluster.testFailover:156->Object.wait:502->Object.wait:-2 »
TestTimedOut
TestMasterNoCluster.testStopDuringStart:130->Object.wait:502->Object.wait:-2
» TestTimedOut
Tests run: 3, Failures: 0, Errors: 2, Skipped: 0
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch, hbase-15295_v6.patch,
> hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
>
[jira] [Commented] (HBASE-15191) CopyTable and VerifyReplication - Option to specify batch size, versions
[ https://issues.apache.org/jira/browse/HBASE-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216671#comment-15216671 ] Hudson commented on HBASE-15191: SUCCESS: Integrated in HBase-1.3-IT #587 (See [https://builds.apache.org/job/HBase-1.3-IT/587/]) HBASE-15191 CopyTable and VerifyReplication - Option to specify batch (tedyu: rev c7ad42867dbf35c5d07e4e3085edc2fd2145) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CopyTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java > CopyTable and VerifyReplication - Option to specify batch size, versions > > > Key: HBASE-15191 > URL: https://issues.apache.org/jira/browse/HBASE-15191 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 0.98.16.1 >Reporter: Parth Shah >Assignee: Parth Shah >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE_15191.patch, HBASE_15191.patch > > > Need option to specify batch size for CopyTable and VerifyReplication. We > are working on patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15265) Implement an asynchronous FSHLog
[
https://issues.apache.org/jira/browse/HBASE-15265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216662#comment-15216662
]
stack commented on HBASE-15265:
---
It took me a while but a 2B run on 9-node cluster passes with this enabled (on
Master -- I got Master to pass ITBLL... in this case Master is not carrying
regions... not sure that related). Will try some more runs.
I see this stuff in the log:
511740 2016-03-29 10:00:05,239 INFO [AsyncFSWAL-pool3-t1] wal.AbstractFSWAL:
Slow sync cost: 483 ms, current pipeline:
[DatanodeInfoWithStorage[10.17.240.22:50031,DS-c70edb05-4a60-4d80-a8b7-5783f707a9da,DISK],
DatanodeInfoWithStorage[10.17.240.27:50031,DS-6539429a-1441-45e8-9d47-6732499f4709,DISK],
DatanodeInfoWithStorage[10.17.240.25:50031,DS-b3225e5d-2c3a-4cf3-a91d-10b345451c24,DISK]]
511741 2016-03-29 10:00:05,239 INFO [AsyncFSWAL-pool3-t1] wal.AbstractFSWAL:
Slow sync cost: 482 ms, current pipeline:
[DatanodeInfoWithStorage[10.17.240.22:50031,DS-c70edb05-4a60-4d80-a8b7-5783f707a9da,DISK],
DatanodeInfoWithStorage[10.17.240.27:50031,DS-6539429a-1441-45e8-9d47-6732499f4709,DISK],
DatanodeInfoWithStorage[10.17.240.25:50031,DS-b3225e5d-2c3a-4cf3-a91d-10b345451c24,DISK]]
511742 2016-03-29 10:00:05,239 INFO [AsyncFSWAL-pool3-t1] wal.AbstractFSWAL:
Slow sync cost: 475 ms, current pipeline:
[DatanodeInfoWithStorage[10.17.240.22:50031,DS-c70edb05-4a60-4d80-a8b7-5783f707a9da,DISK],
DatanodeInfoWithStorage[10.17.240.27:50031,DS-6539429a-1441-45e8-9d47-6732499f4709,DISK],
DatanodeInfoWithStorage[10.17.240.25:50031,DS-b3225e5d-2c3a-4cf3-a91d-10b345451c24,DISK]]
Seems like long time to sync but I see these w/o Async enabled... so just
HDFS... to be investigated.
Or these...
{code}
511541 2016-03-29 09:59:54,779 INFO [AsyncFSWAL-pool3-t1] wal.AbstractFSWAL:
Slow sync cost: 761 ms, current pipeline:
[DatanodeInfoWithStorage[10.17.240.22:50031,DS-59aa4068-698e-4c2d-8f37-d921d8cbd3ec,DISK],
DatanodeInfoWithStorage[10.17.240.27:50031,DS-42e19728-db67-4b51-9cd2-10a3b640a8e5,DISK],
DatanodeInfoWithStorage[10.17.240.25:50031,DS-188eee65-528f-4ced-84ac-894d634590f7,DISK]]
511542 2016-03-29 09:59:54,779 INFO [AsyncFSWAL-pool3-t1] wal.AbstractFSWAL:
Slow sync cost: 760 ms, current pipeline:
[DatanodeInfoWithStorage[10.17.240.22:50031,DS-59aa4068-698e-4c2d-8f37-d921d8cbd3ec,DISK],
DatanodeInfoWithStorage[10.17.240.27:50031,DS-42e19728-db67-4b51-9cd2-10a3b640a8e5,DISK],
DatanodeInfoWithStorage[10.17.240.25:50031,DS-188eee65-528f-4ced-84ac-894d634590f7,DISK]]
511543 2016-03-29 09:59:54,779 INFO [AsyncFSWAL-pool3-t1] wal.AbstractFSWAL:
Slow sync cost: 760 ms, current pipeline:
[DatanodeInfoWithStorage[10.17.240.22:50031,DS-59aa4068-698e-4c2d-8f37-d921d8cbd3ec,DISK],
DatanodeInfoWithStorage[10.17.240.27:50031,DS-42e19728-db67-4b51-9cd2-10a3b640a8e5,DISK],
DatanodeInfoWithStorage[10.17.240.25:50031,DS-188eee65-528f-4ced-84ac-894d634590f7,DISK]]
{code}
... need to dig in on the above.
File issue to make this the default?
Meantime I'll run some more tests.
Nice work [~Apache9]
> Implement an asynchronous FSHLog
>
>
> Key: HBASE-15265
> URL: https://issues.apache.org/jira/browse/HBASE-15265
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-15265-v1.patch, HBASE-15265-v2.patch,
> HBASE-15265-v3.patch, HBASE-15265-v4.patch, HBASE-15265-v5.patch,
> HBASE-15265-v6.patch, HBASE-15265-v7.patch, HBASE-15265-v8.patch,
> HBASE-15265-v8.patch, HBASE-15265-v8.patch, HBASE-15265-v8.patch,
> HBASE-15265-v8.patch, HBASE-15265-v8.patch, HBASE-15265-v8.patch,
> HBASE-15265-v8.patch, HBASE-15265.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15506) FSDataOutputStream.write() allocates new byte buffer on each operation
[ https://issues.apache.org/jira/browse/HBASE-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1521#comment-1521 ] Vladimir Rodionov commented on HBASE-15506: --- Linked HBASE-15506 as well. > FSDataOutputStream.write() allocates new byte buffer on each operation > -- > > Key: HBASE-15506 > URL: https://issues.apache.org/jira/browse/HBASE-15506 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Deep inside stack trace in DFSOutputStream.createPacket. > This should be opened in HDFS. This JIRA is to track HDFS work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Description: LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and recency of the working set. It achieves concurrency by using an O( n ) background thread to prioritize the entries and evict. Accessing an entry is O(1) by a hash table lookup, recording its logical access time, and setting a frequency flag. A write is performed in O(1) time by updating the hash table and triggering an async eviction thread. This provides ideal concurrency and minimizes the latencies by penalizing the thread instead of the caller. However the policy does not age the frequencies and may not be resilient to various workload patterns. W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the frequency in a counting sketch, ages periodically by halving the counters, and orders entries by SLRU. An entry is discarded by comparing the frequency of the new arrival (candidate) to the SLRU's victim, and keeping the one with the highest frequency. This allows the operations to be performed in O(1) time and, though the use of a compact sketch, a much larger history is retained beyond the current working set. In a variety of real world traces the policy had [near optimal hit rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. Concurrency is achieved by buffering and replaying the operations, similar to a write-ahead log. A read is recorded into a striped ring buffer and writes to a queue. The operations are applied in batches under a try-lock by an asynchronous thread, thereby track the usage pattern without incurring high latencies ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). In YCSB benchmarks the results were inconclusive. For a large cache (99% hit rates) the two caches have near identical throughput and latencies with LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4% hit rate improvement and therefore lower latencies. The lack luster result is because a synthetic Zipfian distribution is used, which SLRU performs optimally. In a more varied, real-world workload we'd expect to see improvements by being able to make smarter predictions. The provided patch implements BlockCache using the [Caffeine|https://github.com/ben-manes/caffeine] caching library (see HighScalability [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). Edward Bortnikov and Eshcar Hillel have graciously provided guidance for evaluating this patch ([github branch|https://github.com/ben-manes/hbase/tree/tinylfu]). was: LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and recency of the working set. It achieves concurrency by using an O( n ) background thread to prioritize the entries and evict. Accessing an entry is O(1) by a hash table lookup, recording its logical access time, and setting a frequency flag. A write is performed in O(1) time by updating the hash table and triggering an async eviction thread. This provides ideal concurrency and minimizes the latencies by penalizing the thread instead of the caller. However the policy does not age the frequencies and may not be resilient to various workload patterns. W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the frequency in a counting sketch, ages periodically by halving the counters, and orders entries by SLRU. An entry is discarded by comparing the frequency of the new arrival (candidate) to the SLRU's victim, and keeping the one with the highest frequency. This allows the operations to be performed in O(1) time and, though the use of a compact sketch, a much larger history is retained beyond the current working set. In a variety of real world traces the policy had [near optimal hit rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. Concurrency is achieved by buffering and replaying the operations, similar to a write-ahead log. A read is recorded into a striped ring buffer and writes to a queue. The operations are applied in batches under a try-lock by an asynchronous thread, thereby track the usage pattern without incurring high latencies ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). In YCSB benchmarks the results were inconclusive. For a large cache (99% hit rates) the two caches have near identical throughput and latencies with LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4% hit rate improvement and therefore lower latencies. The lack luster result is because a synthetic Zipfian distribution is used, which SLRU performs optimally. In a more varied, real-world workload we'd expect to see improvements by being able to make smarter predictions. The provided patch implements
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[
https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ben Manes updated HBASE-15560:
--
Description:
LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
recency of the working set. It achieves concurrency by using an O( n )
background thread to prioritize the entries and evict. Accessing an entry is
O(1) by a hash table lookup, recording its logical access time, and setting a
frequency flag. A write is performed in O(1) time by updating the hash table
and triggering an async eviction thread. This provides ideal concurrency and
minimizes the latencies by penalizing the thread instead of the caller. However
the policy does not age the frequencies and may not be resilient to various
workload patterns.
W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
frequency in a counting sketch, ages periodically by halving the counters, and
orders entries by SLRU. An entry is discarded by comparing the frequency of the
new arrival (candidate) to the SLRU's victim, and keeping the one with the
highest frequency. This allows the operations to be performed in O(1) time and,
though the use of a compact sketch, a much larger history is retained beyond
the current working set. In a variety of real world traces the policy had [near
optimal hit rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
Concurrency is achieved by buffering and replaying the operations, similar to a
write-ahead log. A
read is recorded into a striped ring buffer and writes to a queue. The
operations are applied in
batches under a try-lock by an asynchronous thread, thereby track the usage
pattern without incurring high latencies
([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
In YCSB benchmarks the results were inconclusive. For a large cache (99% hit
rates) the two caches have near identical throughput and latencies with
LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4%
hit rate improvement and therefore lower latencies. The lack luster result is
because a synthetic Zipfian distribution is used, which SLRU performs
optimally. In a more varied, real-world workload we'd expect to see
improvements by being able to make smarter predictions.
The provided patch implements BlockCache using the
[Caffeine|https://github.com/ben-manes/caffeine] caching library (see
HighScalability
[article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]).
Edward Bortnikov and Eshcar Hillel have graciously provided guidance for
evaluating this patch ([github
branch|https://github.com/ben-manes/hbase/tree/tinylfu].
was:
LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
recency of the working set. It achieves concurrency by using an
{noformat}O(n){noformat} background thread to prioritize the entries and evict.
Accessing an entry is O(1) by a hash table lookup, recording its logical access
time, and setting a frequency flag. A write is performed in O(1) time by
updating the hash table and triggering an async eviction thread. This provides
ideal concurrency and minimizes the latencies by penalizing the thread instead
of the caller. However the policy does not age the frequencies and may not be
resilient to various workload patterns.
W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
frequency in a counting sketch, ages periodically by halving the counters, and
orders entries by SLRU. An entry is discarded by comparing the frequency of the
new arrival (candidate) to the SLRU's victim, and keeping the one with the
highest frequency. This allows the operations to be performed in O(1) time and,
though the use of a compact sketch, a much larger history is retained beyond
the current working set. In a variety of real world traces the policy had [near
optimal hit rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
Concurrency is achieved by buffering and replaying the operations, similar to a
write-ahead log. A
read is recorded into a striped ring buffer and writes to a queue. The
operations are applied in
batches under a try-lock by an asynchronous thread, thereby track the usage
pattern without incurring high latencies
([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
In YCSB benchmarks the results were inconclusive. For a large cache (99% hit
rates) the two caches have near identical throughput and latencies with
LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4%
hit rate improvement and therefore lower latencies. The lack luster result is
because a synthetic Zipfian distribution is used, which SLRU performs
optimally. In a more varied, real-world workload we'd expect to see
improvements by being able to make smarter predictions.
The provided patch
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[
https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ben Manes updated HBASE-15560:
--
Description:
LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
recency of the working set. It achieves concurrency by using an
{noformat}O(n){noformat} background thread to prioritize the entries and evict.
Accessing an entry is O(1) by a hash table lookup, recording its logical access
time, and setting a frequency flag. A write is performed in O(1) time by
updating the hash table and triggering an async eviction thread. This provides
ideal concurrency and minimizes the latencies by penalizing the thread instead
of the caller. However the policy does not age the frequencies and may not be
resilient to various workload patterns.
W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
frequency in a counting sketch, ages periodically by halving the counters, and
orders entries by SLRU. An entry is discarded by comparing the frequency of the
new arrival (candidate) to the SLRU's victim, and keeping the one with the
highest frequency. This allows the operations to be performed in O(1) time and,
though the use of a compact sketch, a much larger history is retained beyond
the current working set. In a variety of real world traces the policy had [near
optimal hit rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
Concurrency is achieved by buffering and replaying the operations, similar to a
write-ahead log. A
read is recorded into a striped ring buffer and writes to a queue. The
operations are applied in
batches under a try-lock by an asynchronous thread, thereby track the usage
pattern without incurring high latencies
([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
In YCSB benchmarks the results were inconclusive. For a large cache (99% hit
rates) the two caches have near identical throughput and latencies with
LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4%
hit rate improvement and therefore lower latencies. The lack luster result is
because a synthetic Zipfian distribution is used, which SLRU performs
optimally. In a more varied, real-world workload we'd expect to see
improvements by being able to make smarter predictions.
The provided patch implements BlockCache using the
[Caffeine|https://github.com/ben-manes/caffeine] caching library (see
HighScalability
[article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]).
Edward Bortnikov and Eshcar Hillel have graciously provided guidance for
evaluating this patch ([github
branch|https://github.com/ben-manes/hbase/tree/tinylfu].
was:
LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
recency of the working set. It achieves concurrency by using an O(n) background
thread to prioritize the entries and evict. Accessing an entry is O(1) by a
hash table lookup, recording its logical access time, and setting a frequency
flag. A write is performed in O(1) time by updating the hash table and
triggering an async eviction thread. This provides ideal concurrency and
minimizes the latencies by penalizing the thread instead of the caller. However
the policy does not age the frequencies and may not be resilient to various
workload patterns.
W-TinyLFU ([research paper|W-TinyLFU: http://arxiv.org/pdf/1512.00727.pdf])
records the frequency in a counting sketch, ages periodically by halving the
counters, and orders entries by SLRU. An entry is discarded by comparing the
frequency of the new arrival (candidate) to the SLRU's victim, and keeping the
one with the highest frequency. This allows the operations to be performed in
O(1) time and, though the use of a compact sketch, a much larger history is
retained beyond the current working set. In a variety of real world traces the
policy had [near optimal hit
rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
Concurrency is achieved by buffering and replaying the operations, similar to a
write-ahead log. A
read is recorded into a striped ring buffer and writes to a queue. The
operations are applied in
batches under a try-lock by an asynchronous thread, thereby track the usage
pattern without incurring high latencies
([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
In YCSB benchmarks the results were inconclusive. For a large cache (99% hit
rates) the two caches have near identical throughput and latencies with
LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4%
hit rate improvement and therefore lower latencies. The lack luster result is
because a synthetic Zipfian distribution is used, which SLRU performs
optimally. In a more varied, real-world workload we'd expect to see
improvements by being able to make smarter predictions.
The
[jira] [Updated] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Manes updated HBASE-15560: -- Attachment: tinylfu.patch > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Reporter: Ben Manes > Attachments: tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O(n) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|W-TinyLFU: http://arxiv.org/pdf/1512.00727.pdf]) > records the frequency in a counting sketch, ages periodically by halving the > counters, and orders entries by SLRU. An entry is discarded by comparing the > frequency of the new arrival (candidate) to the SLRU's victim, and keeping > the one with the highest frequency. This allows the operations to be > performed in O(1) time and, though the use of a compact sketch, a much larger > history is retained beyond the current working set. In a variety of real > world traces the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A > read is recorded into a striped ring buffer and writes to a queue. The > operations are applied in > batches under a try-lock by an asynchronous thread, thereby track the usage > pattern without incurring high latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15560) TinyLFU-based BlockCache
Ben Manes created HBASE-15560: - Summary: TinyLFU-based BlockCache Key: HBASE-15560 URL: https://issues.apache.org/jira/browse/HBASE-15560 Project: HBase Issue Type: Improvement Components: BlockCache Reporter: Ben Manes LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and recency of the working set. It achieves concurrency by using an O(n) background thread to prioritize the entries and evict. Accessing an entry is O(1) by a hash table lookup, recording its logical access time, and setting a frequency flag. A write is performed in O(1) time by updating the hash table and triggering an async eviction thread. This provides ideal concurrency and minimizes the latencies by penalizing the thread instead of the caller. However the policy does not age the frequencies and may not be resilient to various workload patterns. W-TinyLFU ([research paper|W-TinyLFU: http://arxiv.org/pdf/1512.00727.pdf]) records the frequency in a counting sketch, ages periodically by halving the counters, and orders entries by SLRU. An entry is discarded by comparing the frequency of the new arrival (candidate) to the SLRU's victim, and keeping the one with the highest frequency. This allows the operations to be performed in O(1) time and, though the use of a compact sketch, a much larger history is retained beyond the current working set. In a variety of real world traces the policy had [near optimal hit rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. Concurrency is achieved by buffering and replaying the operations, similar to a write-ahead log. A read is recorded into a striped ring buffer and writes to a queue. The operations are applied in batches under a try-lock by an asynchronous thread, thereby track the usage pattern without incurring high latencies ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). In YCSB benchmarks the results were inconclusive. For a large cache (99% hit rates) the two caches have near identical throughput and latencies with LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a 1-4% hit rate improvement and therefore lower latencies. The lack luster result is because a synthetic Zipfian distribution is used, which SLRU performs optimally. In a more varied, real-world workload we'd expect to see improvements by being able to make smarter predictions. The provided patch implements BlockCache using the [Caffeine|https://github.com/ben-manes/caffeine] caching library (see HighScalability [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). Edward Bortnikov and Eshcar Hillel have graciously provided guidance for evaluating this patch ([github branch|https://github.com/ben-manes/hbase/tree/tinylfu]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15499) Add multiple data type support for increment
[
https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216631#comment-15216631
]
Hadoop QA commented on HBASE-15499:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 45s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 36s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 1s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 44s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 35s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 44s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 35s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 41s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 41s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 36s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-common in the patch
[jira] [Commented] (HBASE-15362) Compression Algorithm does not respect config params from hbase-site
[ https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216605#comment-15216605 ] deepankar commented on HBASE-15362: --- Sure will create a jira > Compression Algorithm does not respect config params from hbase-site > > > Key: HBASE-15362 > URL: https://issues.apache.org/jira/browse/HBASE-15362 > Project: HBase > Issue Type: Sub-task >Reporter: deepankar >Assignee: deepankar >Priority: Trivial > Attachments: HBASE-15362.patch > > > Compression creates conf using new Configuration() and this leads to it not > respecting the confs set in hbase-site, fixing it is trivial using > HBaseConfiguration.create() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15559) BaseMasterAndRegionObserver doesn't implement all the methods
[ https://issues.apache.org/jira/browse/HBASE-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216606#comment-15216606 ] Matteo Bertozzi commented on HBASE-15559: - +1 on v2 > BaseMasterAndRegionObserver doesn't implement all the methods > - > > Key: HBASE-15559 > URL: https://issues.apache.org/jira/browse/HBASE-15559 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15559-v1.patch, HBASE-15559-v2.patch, > HBASE-15559.patch > > > It's supposed to be a class that allows someone to derive from that class and > only need to implement the desired methods. However two methods aren't > implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15559) BaseMasterAndRegionObserver doesn't implement all the methods
[ https://issues.apache.org/jira/browse/HBASE-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216601#comment-15216601 ] Gary Helmling commented on HBASE-15559: --- +1 on v2 > BaseMasterAndRegionObserver doesn't implement all the methods > - > > Key: HBASE-15559 > URL: https://issues.apache.org/jira/browse/HBASE-15559 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15559-v1.patch, HBASE-15559-v2.patch, > HBASE-15559.patch > > > It's supposed to be a class that allows someone to derive from that class and > only need to implement the desired methods. However two methods aren't > implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15559) BaseMasterAndRegionObserver doesn't implement all the methods
[ https://issues.apache.org/jira/browse/HBASE-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15559: -- Attachment: HBASE-15559-v2.patch > BaseMasterAndRegionObserver doesn't implement all the methods > - > > Key: HBASE-15559 > URL: https://issues.apache.org/jira/browse/HBASE-15559 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15559-v1.patch, HBASE-15559-v2.patch, > HBASE-15559.patch > > > It's supposed to be a class that allows someone to derive from that class and > only need to implement the desired methods. However two methods aren't > implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15295) MutateTableAccess.multiMutate() does not get high priority causing a deadlock
[
https://issues.apache.org/jira/browse/HBASE-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216592#comment-15216592
]
Enis Soztutar commented on HBASE-15295:
---
{code}
mvn clean test -Dtest=TestMasterNoCluster
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.master.TestMasterNoCluster
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.738 sec - in
org.apache.hadoop.hbase.master.TestMasterNoCluster
Results :
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
{code}
> MutateTableAccess.multiMutate() does not get high priority causing a deadlock
> -
>
> Key: HBASE-15295
> URL: https://issues.apache.org/jira/browse/HBASE-15295
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.5
>
> Attachments: hbase-15295_v1.patch, hbase-15295_v1.patch,
> hbase-15295_v2.patch, hbase-15295_v3.patch, hbase-15295_v4.patch,
> hbase-15295_v5.patch, hbase-15295_v5.patch, hbase-15295_v6.patch,
> hbase-15295_v6.patch
>
>
> We have seen this in a cluster with Phoenix secondary indexes leading to a
> deadlock. All handlers are busy waiting on the index updates to finish:
> {code}
> "B.defaultRpcServer.handler=50,queue=0,port=16020" #91 daemon prio=5
> os_prio=0 tid=0x7f29f64ba000 nid=0xab51 waiting on condition
> [0x7f29a8762000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000124f1d5c8> (a
> com.google.common.util.concurrent.AbstractFuture$Sync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
> at
> org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
> at
> org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:194)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:144)
> at
> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:134)
> at
> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:457)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:406)
> at
> org.apache.phoenix.hbase.index.Indexer.postBatchMutate(Indexer.java:401)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$36.call(RegionCoprocessorHost.java:1006)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1002)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3162)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2801)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2743)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2031)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at
[jira] [Commented] (HBASE-15554) StoreFile$Writer.appendGeneralBloomFilter generates extra KV
[
https://issues.apache.org/jira/browse/HBASE-15554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216593#comment-15216593
]
Vladimir Rodionov commented on HBASE-15554:
---
{quote}
Are you working on this Vladimir Rodionov?
{quote}
[~ramkrishna.s.vasude...@gmail.com], you can reassign it to yourself. I do work
on HBASE-15542, but there are some other subtasks, higher priority.
> StoreFile$Writer.appendGeneralBloomFilter generates extra KV
>
>
> Key: HBASE-15554
> URL: https://issues.apache.org/jira/browse/HBASE-15554
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
>
> Accounts for 10% memory allocation when BloomFilterType is ROWCOL.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15556) need extensible ConsistencyControl interface
[
https://issues.apache.org/jira/browse/HBASE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216588#comment-15216588
]
Hadoop QA commented on HBASE-15556:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
31s {color} | {color:green} branch-1.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} branch-1.0 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} branch-1.0 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} branch-1.0 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} branch-1.0 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 43s
{color} | {color:red} hbase-common in branch-1.0 has 8 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 34s
{color} | {color:red} hbase-server in branch-1.0 has 59 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hbase-common in branch-1.0 failed with JDK v1.8.0.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s
{color} | {color:red} hbase-server in branch-1.0 failed with JDK v1.8.0.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} branch-1.0 passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 35s {color}
| {color:red} hbase-common-jdk1.8.0 with JDK v1.8.0 generated 6 new + 6
unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 35s {color}
| {color:red} hbase-common-jdk1.8.0 with JDK v1.8.0 generated 6 new + 6
unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 59s {color}
| {color:red} hbase-common-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new + 6
unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 59s {color}
| {color:red} hbase-common-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new + 6
unchanged - 0 fixed = 12 total (was 6) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s
{color} | {color:red} hbase-common: patch generated 26 new + 13 unchanged - 0
fixed = 39 total (was 13) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s
{color} | {color:red} hbase-common: patch generated 26 new + 13 unchanged - 0
fixed = 39 total (was 13) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s
{color} | {color:red} hbase-server: patch generated 26 new + 13 unchanged - 0
fixed = 39 total (was 13) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s
{color} | {color:red} hbase-server: patch generated 26 new + 13 unchanged - 0
fixed = 39 total (was 13) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color}
[jira] [Commented] (HBASE-15543) HFileScannerImpl.getCell() allocates cell object on every call
[
https://issues.apache.org/jira/browse/HBASE-15543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216578#comment-15216578
]
Vladimir Rodionov commented on HBASE-15543:
---
{quote}
The compaction scanner also has caching enabled.
{quote}
Why? Must be off.
{quote}
Is the reported 28% creation of new Cell or new Cell reference when the Cell is
being made against a backing BB that is offheap from BucketCache?
{quote}
On heap. Both byte array and object are allocated.
> HFileScannerImpl.getCell() allocates cell object on every call
> --
>
> Key: HBASE-15543
> URL: https://issues.apache.org/jira/browse/HBASE-15543
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>
> This accounts for 28% of memory allocation in compaction thread.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15555) Memory Management
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216539#comment-15216539 ] stack commented on HBASE-1: --- bq. Do you want to raise a JIRA for this? I think the subtasks here would cover it. The phenomeon is disturbing, that though we are creating less garbage, it makes no difference time-wise when GC'ing. Worth monitoring as an independent concern I'd say... as subtask? And similar the issue of SLABs vs G1GC... Need to get over that hurdle too... definitively... Get to the bottom of it. > Memory Management > - > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Umbrella >Reporter: stack > Attachments: download.svg > > > Umbrella issue on memory management. One-stop shop to learn how we're doing > it, tenets, and work to be done. Collect it all in one place so we are > aligned rather than each trying to do it for themselves. > Subtasks include: > + Map of memory allocation and recycling landscape > + Where are we allocating when we could be recycling > + MSLAB and G1GC don't go together? > + Enable MSLAB pool always? > + With offheap allocations, less GC but it runs for longer? > + Better heuristics recycling: e.g. HBASE-15525 OutOfMemory could occur when > using BoundedByteBufferPool during RPC bursts > + See if can do DFSClient buffer reuse: HBASE-15506 > FSDataOutputStream.write() allocates new byte buffer on each operation > What else? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15556) need extensible ConsistencyControl interface
[ https://issues.apache.org/jira/browse/HBASE-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216554#comment-15216554 ] Walter Koetke commented on HBASE-15556: --- You are correct, Sean. The HBaseBlockingWaitStrategy is another optimization we came up with but it's not directly related to this Jira, so I will resubmit with this piece removed and with the other feedback incorporated (missing javadoc, annotations, etc.) > need extensible ConsistencyControl interface > > > Key: HBASE-15556 > URL: https://issues.apache.org/jira/browse/HBASE-15556 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.0.3 >Reporter: Walter Koetke >Assignee: Walter Koetke > Fix For: 1.0.4 > > Attachments: HBASE-15556-branch-1.0.patch > > > The class MultiVersionConsistencyControl should be abstracted into an > interface ConsistencyControl so it can be extended by a configured custom > implementation class, with MultiVersionConsistencyControl as the default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[
https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-15236:
-
Attachment: HBASE-15236-master-v7.patch
rebased, fixed merge errors.
> Inconsistent cell reads over multiple bulk-loaded HFiles
>
>
> Key: HBASE-15236
> URL: https://issues.apache.org/jira/browse/HBASE-15236
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-15236-master-v2.patch,
> HBASE-15236-master-v3.patch, HBASE-15236-master-v4.patch,
> HBASE-15236-master-v5.patch, HBASE-15236-master-v6.patch,
> HBASE-15236-master-v7.patch, HBASE-15236.patch, TestWithSingleHRegion.java,
> tables_data.zip
>
>
> If there are two bulkloaded hfiles in a region with same seqID, same
> timestamps and duplicate keys*, get and scan may return different values for
> a key. Not sure how this would happen, but one of our customer uploaded a
> dataset with 2 files in a single region and both having same bulk load
> timestamp. These files are small ~50M (I couldn't find any setting for max
> file size that could lead to 2 files). The range of keys in two hfiles are
> overlapping to some extent, but not fully (so the two files are because of
> region merge).
> In such a case, depending on file sizes (used in
> StoreFile.Comparators.SEQ_ID), we may get different values for the same cell
> (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r"
> "cf:".
> To replicate this, i ran ImportTsv twice with 2 different csv files but same
> timestamp (-Dimporttsv.timestamp=1). Collected two resulting hfiles in a
> single directory and loaded with LoadIncrementalHFiles. Here are the commands.
> {noformat}
> //CSV files should be in hdfs:///user/hbase/tmp
> sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c:1,c:2,c:3,c:4,c:5,c:6
> -Dimporttsv.separator='|' -Dimporttsv.timestamp=1
> -Dimporttsv.bulk.output=hdfs:///user/hbase/1 t tmp
> {noformat}
> tables_data.zip contains three tables: t, t2, and t3.
> To load these tables and test with them, use the attached
> TestWithSingleHRegion.java. (Change ROOT_DIR and TABLE_NAME variables
> appropriately)
> Hfiles for table 't':
> 1) 07922ac90208445883b2ddc89c424c89 : length = 1094: KVs = (c:5, 55) (c:6, 66)
> 2) 143137fa4fa84625b4e8d1d84a494f08 : length = 1192: KVs = (c:1, 1) (c:2, 2)
> (c:3, 3) (c:4, 4) (c:5, 5) (c:6, 6)
> Get returns 5 where as Scan returns 55.
> On the other hand if I make the first hfile, the one with values 55 and 66
> bigger in size than second hfile, then:
> Get returns 55 where as Scan returns 55.
> This can be seen in table 't2'.
> Weird things:
> 1. Get consistently returns values from larger hfile.
> 2. Scan consistently returns 55.
> Reason:
> Explanation 1: We add StoreFileScanners to heap by iterating over list of
> store files which is kept sorted in DefaultStoreFileManger using
> StoreFile.Comparators.SEQ_ID. It sorts the file first by increasing seq id,
> then by decreasing filesize. So our larger files sizes are always in the
> start.
> Explanation 2: In KeyValueHeap.next(), if both current and this.heap.peek()
> scanners (type is StoreFileScanner in this case) point to KVs which compare
> as equal, we put back the current scanner into heap, and call pollRealKV() to
> get new one. While inserting, KVScannerComparator is used which compares the
> two (one already in heap and the one being inserted) as equal and inserts it
> after the existing one. \[1\]
> Say s1 is scanner over first hfile and s2 over second.
> 1. We scan with s2 to get values 1, 2, 3, 4
> 2. see that both scanners have c:5 as next key, put current scanner s2 back
> in heap, take s1 out
> 4. return 55, advance s1 to key c:6
> 5. compare c:6 to c:5, put back s2 in heap since it has larger key, take out
> s1
> 6. ignore it's value (there is code for that too), advance s2 to c:6
> Go back to step 2 with both scanners having c:6 as next key
> This is wrong because if we add a key between c:4 and c:5 to first hfile, say
> (c:44, foo) which makes s1 as the 'current' scanner when we hit step 2, then
> we'll get the values - 1, 2, 3, 4, foo, 5, 6.
> (try table t3)
> Fix:
> Assign priority ids to StoreFileScanners when inserting them into heap,
> latest hfiles get higher priority, and use them in KVComparator instead of
> just seq id.
> \[1\]
> [PriorityQueue.siftUp()|http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/PriorityQueue.java#586]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15559) BaseMasterAndRegionObserver doesn't implement all the methods
[ https://issues.apache.org/jira/browse/HBASE-15559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216476#comment-15216476 ] Gary Helmling commented on HBASE-15559: --- +1 on removing the abstract. I would say to do the same for BaseRegionObserver as well. > BaseMasterAndRegionObserver doesn't implement all the methods > - > > Key: HBASE-15559 > URL: https://issues.apache.org/jira/browse/HBASE-15559 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15559-v1.patch, HBASE-15559.patch > > > It's supposed to be a class that allows someone to derive from that class and > only need to implement the desired methods. However two methods aren't > implemented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13639) SyncTable - rsync for HBase tables
[ https://issues.apache.org/jira/browse/HBASE-13639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216471#comment-15216471 ] Abhishek Soni commented on HBASE-13639: --- Thanks [~davelatham], It is just what I needed. > SyncTable - rsync for HBase tables > -- > > Key: HBASE-13639 > URL: https://issues.apache.org/jira/browse/HBASE-13639 > Project: HBase > Issue Type: New Feature > Components: mapreduce, Operability, tooling >Reporter: Dave Latham >Assignee: Dave Latham > Labels: tooling > Fix For: 2.0.0, 0.98.14, 1.2.0 > > Attachments: HBASE-13639-0.98-addendum-hadoop-1.patch, > HBASE-13639-0.98.patch, HBASE-13639-v1.patch, HBASE-13639-v2.patch, > HBASE-13639-v3-0.98.patch, HBASE-13639-v3.patch, HBASE-13639.patch > > > Given HBase tables in remote clusters with similar but not identical data, > efficiently update a target table such that the data in question is identical > to a source table. Efficiency in this context means using far less network > traffic than would be required to ship all the data from one cluster to the > other. Takes inspiration from rsync. > Design doc: > https://docs.google.com/document/d/1-2c9kJEWNrXf5V4q_wBcoIXfdchN7Pxvxv1IO6PW0-U/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15499) Add multiple data type support for increment
[
https://issues.apache.org/jira/browse/HBASE-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216441#comment-15216441
]
Hadoop QA commented on HBASE-15499:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 46s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 55s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 22s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 8s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 9s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 8s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 9s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
43m 1s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 4s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 36s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 5s
{color} | {color:green} hbase-common in the patch
[jira] [Commented] (HBASE-15538) Implement secure async protobuf wal writer
[
https://issues.apache.org/jira/browse/HBASE-15538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15216418#comment-15216418
]
Hadoop QA commented on HBASE-15538:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 16s
{color} | {color:red} hbase-server: patch generated 1 new + 26 unchanged - 3
fixed = 27 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 14s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 19s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12795854/HBASE-15538-v2.patch |
| JIRA Issue | HBASE-15538 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux pietas.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT
Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7f39baf |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| findbugs | v3.0.0 |
| checkstyle |
