[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521378#comment-16521378 ] stack commented on HBASE-20781: --- See attached examples [^2.0621.2.12782.cpu.svg] and [^2.0621.256k.51351.cpu.svg] ; see how we have above doMiniBatchMutate, a dedicated doWALAppend CPU spike. See how all it is doing is FSWALEntry#collectFamilies. > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20781: -- Attachment: 2.0621.256k.51351.cpu.svg > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20781: -- Attachment: 2.0621.2.12782.cpu.svg > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
stack created HBASE-20781: - Summary: Save recalculating families in a WALEdit batch of Cells Key: HBASE-20781 URL: https://issues.apache.org/jira/browse/HBASE-20781 Project: HBase Issue Type: Sub-task Components: Performance Reporter: stack Assignee: stack Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit Cells belong to up front but down after the RingBuffer when we make an FSWALEdit, we spin through all the Cells again to figure the set of families in a particularly painful way. Just pass the calculated family set in the WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20236) [locking] Write-time worst offenders
[
https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521376#comment-16521376
]
stack commented on HBASE-20236:
---
Attached latest (after HBASE-20780). [^2.0623.111354.lock.svg]
> [locking] Write-time worst offenders
>
>
> Key: HBASE-20236
> URL: https://issues.apache.org/jira/browse/HBASE-20236
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Priority: Major
> Attachments: 2.0623.111354.lock.svg, HBASE-20236.master.001.patch,
> no_semaphore_vs_semaphore.png
>
>
> Messing w/ my new toy, here are worst offenders locking; they must be bad if
> they show up in this sampling profiler:
> {code}
> 7 Total: 769321884622 (99.24%) samples: 2965
> 8 [ 0] java.util.concurrent.Semaphore$NonfairSync
> 9 [ 1] sun.misc.Unsafe.park
> 10 [ 2] java.util.concurrent.locks.LockSupport.park
> 11 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 12 [ 4]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
> 13 [ 5]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
> 14 [ 6] java.util.concurrent.Semaphore.acquire
> 15 [ 7]
> org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
> 16 [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 17
> 18 Total: 4284274263 (0.55%) samples: 23543
> 19 [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
> 20 [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
> 21 [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
> 22 [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
> 23 [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
> 24 [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
> 25 [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 26 [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 27 [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
> 28 [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
> 29 [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
> 30 [11]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
> 31 [12]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
> 32 [13]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
> 33 [14]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
> 34 [15]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
> 35 [16]
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
> 36 [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
> 37 [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 38 [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 39 [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
> 40 [21]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
> 41 [22]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
> 42 [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
> 43 [24]
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
> 44 [25] org.apache.hadoop.hbase.ipc.RpcServer.call
> 45 [26] org.apache.hadoop.hbase.ipc.CallRunner.run
> 46 [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 47 [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 48
> 49 Total: 717708856 (0.09%) samples: 214
> 50 [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
> 51 [ 1] sun.misc.Unsafe.park
> 52 [ 2] java.util.concurrent.locks.LockSupport.park
> 53 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 54 [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
> 55 [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
> 56 [ 6] java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
> 57 [ 7] org.apache.hadoop.hbase.regionserver.HRegion.blockUpdates
> 58 [ 8]
> org.apache.hadoop.hbase.regionserver.RegionServicesForStores.blockUpdates
> 59 [ 9]
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory
> 60 [10]
> org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run
> 61 [11] java.util.concurrent.ThreadPoolExecutor.runWorker
> 62 [12]
[jira] [Updated] (HBASE-20236) [locking] Write-time worst offenders
[
https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20236:
--
Attachment: 2.0623.111354.lock.svg
> [locking] Write-time worst offenders
>
>
> Key: HBASE-20236
> URL: https://issues.apache.org/jira/browse/HBASE-20236
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Priority: Major
> Attachments: 2.0623.111354.lock.svg, HBASE-20236.master.001.patch,
> no_semaphore_vs_semaphore.png
>
>
> Messing w/ my new toy, here are worst offenders locking; they must be bad if
> they show up in this sampling profiler:
> {code}
> 7 Total: 769321884622 (99.24%) samples: 2965
> 8 [ 0] java.util.concurrent.Semaphore$NonfairSync
> 9 [ 1] sun.misc.Unsafe.park
> 10 [ 2] java.util.concurrent.locks.LockSupport.park
> 11 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 12 [ 4]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
> 13 [ 5]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
> 14 [ 6] java.util.concurrent.Semaphore.acquire
> 15 [ 7]
> org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
> 16 [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 17
> 18 Total: 4284274263 (0.55%) samples: 23543
> 19 [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
> 20 [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
> 21 [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
> 22 [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
> 23 [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
> 24 [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
> 25 [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 26 [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 27 [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
> 28 [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
> 29 [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
> 30 [11]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
> 31 [12]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
> 32 [13]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
> 33 [14]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
> 34 [15]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
> 35 [16]
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
> 36 [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
> 37 [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 38 [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 39 [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
> 40 [21]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
> 41 [22]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
> 42 [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
> 43 [24]
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
> 44 [25] org.apache.hadoop.hbase.ipc.RpcServer.call
> 45 [26] org.apache.hadoop.hbase.ipc.CallRunner.run
> 46 [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 47 [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 48
> 49 Total: 717708856 (0.09%) samples: 214
> 50 [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
> 51 [ 1] sun.misc.Unsafe.park
> 52 [ 2] java.util.concurrent.locks.LockSupport.park
> 53 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 54 [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
> 55 [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
> 56 [ 6] java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
> 57 [ 7] org.apache.hadoop.hbase.regionserver.HRegion.blockUpdates
> 58 [ 8]
> org.apache.hadoop.hbase.regionserver.RegionServicesForStores.blockUpdates
> 59 [ 9]
> org.apache.hadoop.hbase.regionserver.CompactingMemStore.flushInMemory
> 60 [10]
> org.apache.hadoop.hbase.regionserver.CompactingMemStore$InMemoryFlushRunnable.run
> 61 [11] java.util.concurrent.ThreadPoolExecutor.runWorker
> 62 [12] java.util.concurrent.ThreadPoolExecutor$Worker.run
> 63 [13] java.lang.Thread.run
> ...
> {code}
--
[jira] [Commented] (HBASE-20236) [locking] Write-time worst offenders
[
https://issues.apache.org/jira/browse/HBASE-20236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521375#comment-16521375
]
stack commented on HBASE-20236:
---
So, MutableSegment locking has been cleaned up in HBASE-20411. The fast path
shows as a locking offender still but I just tried removing it again and again,
there is a slow down if it is turned off; it looks bad in the locking profile
but our throughput benefits. Would be interested if it could be made run
smoother. Let me attach a graph of current write lock offenders. Mostly reading
and writing... off the socket, mvcc, and of course the rpc scheduling semaphore
(20%).
> [locking] Write-time worst offenders
>
>
> Key: HBASE-20236
> URL: https://issues.apache.org/jira/browse/HBASE-20236
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.0-beta-2
>Reporter: stack
>Priority: Major
> Attachments: HBASE-20236.master.001.patch,
> no_semaphore_vs_semaphore.png
>
>
> Messing w/ my new toy, here are worst offenders locking; they must be bad if
> they show up in this sampling profiler:
> {code}
> 7 Total: 769321884622 (99.24%) samples: 2965
> 8 [ 0] java.util.concurrent.Semaphore$NonfairSync
> 9 [ 1] sun.misc.Unsafe.park
> 10 [ 2] java.util.concurrent.locks.LockSupport.park
> 11 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 12 [ 4]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly
> 13 [ 5]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly
> 14 [ 6] java.util.concurrent.Semaphore.acquire
> 15 [ 7]
> org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner
> 16 [ 8] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 17
> 18 Total: 4284274263 (0.55%) samples: 23543
> 19 [ 0] org.apache.hadoop.hbase.regionserver.MutableSegment
> 20 [ 1] org.apache.hadoop.hbase.ByteBufferKeyValue.getSequenceId
> 21 [ 2] org.apache.hadoop.hbase.regionserver.Segment.updateMetaInfo
> 22 [ 3] org.apache.hadoop.hbase.regionserver.Segment.internalAdd
> 23 [ 4] org.apache.hadoop.hbase.regionserver.MutableSegment.add
> 24 [ 5] org.apache.hadoop.hbase.regionserver.AbstractMemStore.internalAdd
> 25 [ 6] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 26 [ 7] org.apache.hadoop.hbase.regionserver.AbstractMemStore.add
> 27 [ 8] org.apache.hadoop.hbase.regionserver.HStore.add
> 28 [ 9] org.apache.hadoop.hbase.regionserver.HRegion.applyToMemStore
> 29 [10] org.apache.hadoop.hbase.regionserver.HRegion.access$600
> 30 [11]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.applyFamilyMapToMemStore
> 31 [12]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.lambda$writeMiniBatchOperationsToMemStore$0
> 32 [13]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation$$Lambda$442.1445825895.visit
> 33 [14]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations
> 34 [15]
> org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.writeMiniBatchOperationsToMemStore
> 35 [16]
> org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.writeMiniBatchOperationsToMemStore
> 36 [17] org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate
> 37 [18] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 38 [19] org.apache.hadoop.hbase.regionserver.HRegion.batchMutate
> 39 [20] org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp
> 40 [21]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp
> 41 [22]
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation
> 42 [23] org.apache.hadoop.hbase.regionserver.RSRpcServices.multi
> 43 [24]
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod
> 44 [25] org.apache.hadoop.hbase.ipc.RpcServer.call
> 45 [26] org.apache.hadoop.hbase.ipc.CallRunner.run
> 46 [27] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 47 [28] org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run
> 48
> 49 Total: 717708856 (0.09%) samples: 214
> 50 [ 0] java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
> 51 [ 1] sun.misc.Unsafe.park
> 52 [ 2] java.util.concurrent.locks.LockSupport.park
> 53 [ 3]
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt
> 54 [ 4] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued
> 55 [ 5] java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire
> 56 [ 6] java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock
> 57 [ 7]
[jira] [Commented] (HBASE-20711) Save on a Cell iteration when writing
[
https://issues.apache.org/jira/browse/HBASE-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521370#comment-16521370
]
stack commented on HBASE-20711:
---
bq. Do you mean the path using the CellScanner (see below)?
No. I meant this:
{code}
BTW, why doing the following change? The change makes us copy the row many
times. Is it related another bugfix boss?
- .setRow(put.getRow())
+ cell = cellBuilder.clear()
+ .setRow(proto.getRow().toByteArray())
{code}
I was afraid that the row in the Put might not match what was in the Cell. If
it didn't, somehow, then we'd pass in a Cell that was not supplied by the
client?
I like your point that it means we'll make a new row each time... that we
should avoid that if we can.
I see this size check costing us about 3% of CPU when writing. Maybe 1% is the
iteration cost? Less? If it makes the code easier, we should keep it as is?
> Save on a Cell iteration when writing
> -
>
> Key: HBASE-20711
> URL: https://issues.apache.org/jira/browse/HBASE-20711
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Attachments: HBASE-20711.branch-2.0.001.patch
>
>
> This is a minor savings. We were doing a spin through all Cells on receipt
> just to check their size when subsequently, we were doing an iteration of all
> Cells to insert. It manifest as a little spike in perf output. This change
> removes the purposed spin through Cells and just does size check as part of
> the general Cell insert (perf spike no longer shows but the cost of the size
> check still remains).
> There is also a minor bug fix where on receipt we were using the Puts row
> rather than the Cells row; client may have succeeded in submitting a Cell
> that disagreed with the hosting Mutation and it would have been written as
> something else altogether -- with the Puts row -- rather than being rejected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521367#comment-16521367
]
stack commented on HBASE-20710:
---
Pushed a checkstyle addendum to branch-2.0+
commit 959d9c16f305fbed0298e6da4fab43f4c1a5e478 (HEAD -> m, origin/master,
origin/HEAD)
Author: Michael Stack
Date: Sat Jun 23 20:41:47 2018 -0700
HBASE-20710 extra cloneFamily() in Mutation.add(Cell); ADDENDUM checkstyle
diff --git
a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
index 3d935011dd..dc8199df72 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
@@ -115,10 +115,10 @@ public abstract class Mutation extends
OperationWithAttributes implements Row, C
super(clone);
this.row = clone.getRow();
this.ts = clone.getTimestamp();
-this.familyMap = clone.getFamilyCellMap().entrySet().stream()
- .collect(Collectors.toMap(e -> e.getKey(), e -> new
ArrayList<>(e.getValue()), (k, v) -> {
-throw new RuntimeException("collisions!!!");
- }, () -> new TreeMap<>(Bytes.BYTES_COMPARATOR)));
+this.familyMap = clone.getFamilyCellMap().entrySet().stream().
+ collect(Collectors.toMap(e -> e.getKey(), e -> new
ArrayList<>(e.getValue()), (k, v) -> {
+throw new RuntimeException("collisions!!!");
+ }, () -> new TreeMap<>(Bytes.BYTES_COMPARATOR)));
}
> extra cloneFamily() in Mutation.add(Cell)
> -
>
> Key: HBASE-20710
> URL: https://issues.apache.org/jira/browse/HBASE-20710
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Affects Versions: 2.0.1
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg,
> HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch,
> HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch,
> alloc-put-fix.svg, alloc-put-orig.svg
>
>
> The cpu profiling shows that during PE randomWrite testing, about 1 percent
> of time is spent in cloneFamily. Reviewing code found that when a cell is DBB
> backed ByteBuffKeyValueCell (which is default with Netty Rpc),
> cell.getFamilyArray() will call cloneFamily() and there is again a
> cloneFamily() in the following line of the code. since this is the critical
> write path processing, this needs to be optimized.
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20710: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.1.0 3.0.0 Status: Resolved (was: Patch Available) Pushed to branch-2.0, 2.1, and master. Thanks [~huaxiang] I pushed it because want it in my profiling runs. Hope you don't mind. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg, > HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch, > HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Status: Patch Available (was: Open) > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg, > 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch, > HBASE-20780.branch-2.0.001.patch > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521360#comment-16521360 ] stack commented on HBASE-20780: --- .001 cleans up logging. Removes an instance. Aggregates all logging in one line. See how it seems to remove logging as a lock hump in posted profile [^2.0623.111354.lock.svg] > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg, > 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch, > HBASE-20780.branch-2.0.001.patch > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: HBASE-20780.branch-2.0.001.patch > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg, > 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch, > HBASE-20780.branch-2.0.001.patch > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: 2.0623.111354.lock.svg > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg, > 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch, > HBASE-20780.branch-2.0.001.patch > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: HBASE-20780.branch-2.0.001.patch > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg, > HBASE-20780.branch-2.0.001.patch > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521356#comment-16521356 ] stack commented on HBASE-20780: --- You can see how logging is the most prominent peek in the two profiles posted.. .one with defaults [^2.0621.2.12782.lock.svg] and second with 2MB async WAL buffer size [^2.0621.2M.46340.lock.svg] (especially bad in second case). > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: 2.0621.2M.46340.lock.svg > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: (was: 1.0621.2.4007.lock.svg) > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: (was: 2.0621.512k.65671.lock.svg) > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: 2.0621.2.12782.lock.svg > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: 1.0621.2.4007.lock.svg > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Attachment: 2.0621.512k.65671.lock.svg > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.lock.svg > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20780) ServerRpcConnection logging cleanup
stack created HBASE-20780: - Summary: ServerRpcConnection logging cleanup Key: HBASE-20780 URL: https://issues.apache.org/jira/browse/HBASE-20780 Project: HBase Issue Type: Sub-task Components: logging, Performance Reporter: stack Assignee: stack The logging we do inside in connection header parsing shows as worst offender in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it makes any difference in throughput but lets get it out of the way. Let me load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521353#comment-16521353 ] stack commented on HBASE-20710: --- I tried the patch. Here is before... [^2.0621.2.12782.alloc.svg] and here is with the patch applied [^2.0623.111354.alloc.svg] You can see the two clone families in the first graph above the CallRunner#run hump. Hard to find the single clone family in second graph... its really small. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg, > HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch, > HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20710: -- Attachment: 2.0621.2.12782.alloc.svg > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: 2.0621.2.12782.alloc.svg, 2.0623.111354.alloc.svg, > HBASE-20710-master-v001.patch, HBASE-20710-master-v002.patch, > HBASE-20710-master-v002.patch, HBASE-20710.master.001.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20710: -- Attachment: 2.0623.111354.alloc.svg > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: 2.0623.111354.alloc.svg, HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > HBASE-20710.master.001.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[
https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521351#comment-16521351
]
Hadoop QA commented on HBASE-20710:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} hbase-client: The patch generated 2 new + 23 unchanged
- 0 fixed = 25 total (was 23) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
24s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20710 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928887/HBASE-20710.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cc85110d7180 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 55147c7eae |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13360/artifact/patchprocess/diff-checkstyle-hbase-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13360/testReport/ |
| Max.
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521340#comment-16521340 ] stack commented on HBASE-20710: --- Bit confusing but .001 is rebase (its [~huaxiang]'s .002 fixed so works on master branch) > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > HBASE-20710.master.001.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20710: -- Attachment: HBASE-20710.master.001.patch > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > HBASE-20710.master.001.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20775) TestMultiParallel is flakey
[ https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20775: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to branch-2.1+. > TestMultiParallel is flakey > --- > > Key: HBASE-20775 > URL: https://issues.apache.org/jira/browse/HBASE-20775 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.2.0 > > Attachments: HBASE-20775-addendum.patch, HBASE-20775-v1.patch, > HBASE-20775.patch, HBASE-20775.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > Seems the problem is related to HBASE-20708, where we will give states other > than OPEN to meta region. > Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20777) TestAsyncTableBatch is flakey
[ https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521313#comment-16521313 ] Duo Zhang commented on HBASE-20777: --- https://builds.apache.org/job/HBASE-Flaky-Tests/33593/testReport/junit/org.apache.hadoop.hbase.replication/TestReplicationKillMasterRS/killOneMasterRS/ Seems the same problem. Connecting to an aborted RS leads to connection timed out, not connection refused. So the problem maybe that the abort method for RS does not work correctly. Let me dig more. > TestAsyncTableBatch is flakey > - > > Key: HBASE-20777 > URL: https://issues.apache.org/jira/browse/HBASE-20777 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20777.patch, > org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt > > > The log is very strange, we keep sending request to a dead RS, and the result > is not connection refused, but rpc timeout, and later it becomes > CallQueueTooBig... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20775) TestMultiParallel is flakey
[ https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521310#comment-16521310 ] Duo Zhang commented on HBASE-20775: --- Seems worked. Let me push to other branches. > TestMultiParallel is flakey > --- > > Key: HBASE-20775 > URL: https://issues.apache.org/jira/browse/HBASE-20775 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20775-addendum.patch, HBASE-20775-v1.patch, > HBASE-20775.patch, HBASE-20775.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > Seems the problem is related to HBASE-20708, where we will give states other > than OPEN to meta region. > Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20775) TestMultiParallel is flakey
[
https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521285#comment-16521285
]
Hudson commented on HBASE-20775:
Results for branch master
[build #374 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/374/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/374//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/374//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/374//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> TestMultiParallel is flakey
> ---
>
> Key: HBASE-20775
> URL: https://issues.apache.org/jira/browse/HBASE-20775
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20775-addendum.patch, HBASE-20775-v1.patch,
> HBASE-20775.patch, HBASE-20775.patch,
> org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
>
>
> Seems the problem is related to HBASE-20708, where we will give states other
> than OPEN to meta region.
> Let me dig more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20752) Make sure the regions are truly reopened after ReopenTableRegionsProcedure
[ https://issues.apache.org/jira/browse/HBASE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521264#comment-16521264 ] Josh Elser commented on HBASE-20752: Sorry for the delay from me, Duo. Took a look at what you pushed, but nothing jumped out at me. Will bring it into what we've been testing too. > Make sure the regions are truly reopened after ReopenTableRegionsProcedure > -- > > Key: HBASE-20752 > URL: https://issues.apache.org/jira/browse/HBASE-20752 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: HBASE-20752-branch-2.0.patch, > HBASE-20752-branch-2.0.patch, HBASE-20752-branch-2.0.patch, > HBASE-20752-v1.patch, HBASE-20752.patch > > > The MRP may fail due to some reason, so we need to add some checks in > ReopenTableRegionsProcedure, if we fail to reopen a region, then we should > schedule a new MRP for it again. > And a future improvement maybe that, we can introduce a new type of procedure > to reopen a region, where we just need a simple call to RS, and do not need > to unassign it and assign it again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521253#comment-16521253 ] stack commented on HBASE-20710: --- +1 And +1 for branch-2.0. Why tests fail? Thanks [~huaxiang] > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.2 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, HBASE-20710-master-v002.patch, > alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20557) Backport HBASE-17215 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521230#comment-16521230
]
Hadoop QA commented on HBASE-20557:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
37s{color} | {color:red} hbase-server: The patch generated 1 new + 385
unchanged - 0 fixed = 386 total (was 385) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
2m 0s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 19s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA Issue | HBASE-20557 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928874/HBASE-20557.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a4207d463635 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64
[jira] [Commented] (HBASE-19064) Synchronous replication for HBase
[
https://issues.apache.org/jira/browse/HBASE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521213#comment-16521213
]
Hudson commented on HBASE-19064:
Results for branch HBASE-19064
[build #170 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/170/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/170//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/170//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/170//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Synchronous replication for HBase
> -
>
> Key: HBASE-19064
> URL: https://issues.apache.org/jira/browse/HBASE-19064
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> The guys from Alibaba made a presentation on HBaseCon Asia about the
> synchronous replication for HBase. We(Xiaomi) think this is a very useful
> feature for HBase so we want to bring it into the community version.
> This is a big feature so we plan to do it in a feature branch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20557) Backport HBASE-17215 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20557: - Attachment: HBASE-20557.branch-1.001.patch > Backport HBASE-17215 to branch-1 > > > Key: HBASE-20557 > URL: https://issues.apache.org/jira/browse/HBASE-20557 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20557.branch-1.001.patch > > > As part of HBASE-20555, HBASE-17215 is the second patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20557) Backport HBASE-17215 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu updated HBASE-20557: - Status: Patch Available (was: Open) > Backport HBASE-17215 to branch-1 > > > Key: HBASE-20557 > URL: https://issues.apache.org/jira/browse/HBASE-20557 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20557.branch-1.001.patch > > > As part of HBASE-20555, HBASE-17215 is the second patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20557) Backport HBASE-17215 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521210#comment-16521210 ] Tak Lon (Stephen) Wu commented on HBASE-20557: -- created a [pull request #79|https://github.com/apache/hbase/pull/79] instead of using review board. this is the first time experience so I also attached the patch here for triggering the auto build (is it necessary when using pull request?). > Backport HBASE-17215 to branch-1 > > > Key: HBASE-20557 > URL: https://issues.apache.org/jira/browse/HBASE-20557 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Attachments: HBASE-20557.branch-1.001.patch > > > As part of HBASE-20555, HBASE-17215 is the second patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20777) TestAsyncTableBatch is flakey
[
https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521208#comment-16521208
]
Hadoop QA commented on HBASE-20777:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}143m 51s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.procedure.TestProcedurePriority |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20777 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928872/HBASE-20777.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9a8a2d94b2e9 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 55147c7eae |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/13358/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13358/testReport/ |
| Max. process+thread count | 4707 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521202#comment-16521202 ] stack commented on HBASE-20716: --- We check the unsafe flag on each and every byte read; e.g. method names figuring data lengths > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20752) Make sure the regions are truly reopened after ReopenTableRegionsProcedure
[
https://issues.apache.org/jira/browse/HBASE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521199#comment-16521199
]
Hudson commented on HBASE-20752:
Results for branch branch-2.0
[build #462 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/462/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/462//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/462//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/462//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Make sure the regions are truly reopened after ReopenTableRegionsProcedure
> --
>
> Key: HBASE-20752
> URL: https://issues.apache.org/jira/browse/HBASE-20752
> Project: HBase
> Issue Type: Bug
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: HBASE-20752-branch-2.0.patch,
> HBASE-20752-branch-2.0.patch, HBASE-20752-branch-2.0.patch,
> HBASE-20752-v1.patch, HBASE-20752.patch
>
>
> The MRP may fail due to some reason, so we need to add some checks in
> ReopenTableRegionsProcedure, if we fail to reopen a region, then we should
> schedule a new MRP for it again.
> And a future improvement maybe that, we can introduce a new type of procedure
> to reopen a region, where we just need a simple call to RS, and do not need
> to unassign it and assign it again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20777) TestAsyncTableBatch is flakey
[ https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20777: -- Attachment: HBASE-20777.patch > TestAsyncTableBatch is flakey > - > > Key: HBASE-20777 > URL: https://issues.apache.org/jira/browse/HBASE-20777 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Priority: Major > Attachments: HBASE-20777.patch, > org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt > > > The log is very strange, we keep sending request to a dead RS, and the result > is not connection refused, but rpc timeout, and later it becomes > CallQueueTooBig... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20777) TestAsyncTableBatch is flakey
[ https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521089#comment-16521089 ] Duo Zhang commented on HBASE-20777: --- Set a small rpc timeout to see if it helps... > TestAsyncTableBatch is flakey > - > > Key: HBASE-20777 > URL: https://issues.apache.org/jira/browse/HBASE-20777 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20777.patch, > org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt > > > The log is very strange, we keep sending request to a dead RS, and the result > is not connection refused, but rpc timeout, and later it becomes > CallQueueTooBig... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20777) TestAsyncTableBatch is flakey
[ https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20777: -- Assignee: Duo Zhang Status: Patch Available (was: Open) > TestAsyncTableBatch is flakey > - > > Key: HBASE-20777 > URL: https://issues.apache.org/jira/browse/HBASE-20777 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-20777.patch, > org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt > > > The log is very strange, we keep sending request to a dead RS, and the result > is not connection refused, but rpc timeout, and later it becomes > CallQueueTooBig... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20775) TestMultiParallel is flakey
[
https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521083#comment-16521083
]
Hadoop QA commented on HBASE-20775:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} hbase-server: The patch generated 0 new + 3
unchanged - 6 fixed = 3 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}142m
58s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20775 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928866/HBASE-20775-addendum.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux edc24dc11c12 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 14087cc919 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13357/testReport/ |
| Max. process+thread count | 4594 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13357/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message
[jira] [Commented] (HBASE-20775) TestMultiParallel is flakey
[ https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521049#comment-16521049 ] Duo Zhang commented on HBASE-20775: --- Pushed the addendum to master. Let's see if it works... > TestMultiParallel is flakey > --- > > Key: HBASE-20775 > URL: https://issues.apache.org/jira/browse/HBASE-20775 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20775-addendum.patch, HBASE-20775-v1.patch, > HBASE-20775.patch, HBASE-20775.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > Seems the problem is related to HBASE-20708, where we will give states other > than OPEN to meta region. > Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20769) getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
[
https://issues.apache.org/jira/browse/HBASE-20769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521028#comment-16521028
]
Jingyun Tian commented on HBASE-20769:
--
Got it, thx for your review.
> getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
> ---
>
> Key: HBASE-20769
> URL: https://issues.apache.org/jira/browse/HBASE-20769
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0, 2.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20769.master.001.patch
>
>
> When numSplits > 1, getSplits may create split that has start row smaller
> than user specified scan's start row or stop row larger than user specified
> scan's stop row.
> {code}
> byte[][] sp = sa.split(hri.getStartKey(), hri.getEndKey(), numSplits,
> true);
> for (int i = 0; i < sp.length - 1; i++) {
> if (PrivateCellUtil.overlappingKeys(scan.getStartRow(),
> scan.getStopRow(), sp[i],
> sp[i + 1])) {
> List hosts =
> calculateLocationsForInputSplit(conf, htd, hri, tableDir,
> localityEnabled);
> Scan boundedScan = new Scan(scan);
> boundedScan.setStartRow(sp[i]);
> boundedScan.setStopRow(sp[i + 1]);
> splits.add(new InputSplit(htd, hri, hosts, boundedScan,
> restoreDir));
> }
> }
> {code}
> Since we split keys by the range of regions, when sp[i] < scan.getStartRow()
> or sp[i + 1] > scan.getStopRow(), the created bounded scan may contain range
> that over user defined scan.
> fix should be simple:
> {code}
> boundedScan.setStartRow(
> Bytes.compareTo(scan.getStartRow(), sp[i]) > 0 ? scan.getStartRow() : sp[i]);
> boundedScan.setStopRow(
> Bytes.compareTo(scan.getStopRow(), sp[i + 1]) < 0 ? scan.getStopRow() : sp[i
> + 1]);
> {code}
> I will also try to add UTs to help discover this problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20775) TestMultiParallel is flakey
[ https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20775: -- Attachment: HBASE-20775-addendum.patch > TestMultiParallel is flakey > --- > > Key: HBASE-20775 > URL: https://issues.apache.org/jira/browse/HBASE-20775 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20775-addendum.patch, HBASE-20775-v1.patch, > HBASE-20775.patch, HBASE-20775.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > Seems the problem is related to HBASE-20708, where we will give states other > than OPEN to meta region. > Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20775) TestMultiParallel is flakey
[ https://issues.apache.org/jira/browse/HBASE-20775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521021#comment-16521021 ] Duo Zhang commented on HBASE-20775: --- Thanks [~zghaobac]. Let's disable it. The region on master feature is broken. > TestMultiParallel is flakey > --- > > Key: HBASE-20775 > URL: https://issues.apache.org/jira/browse/HBASE-20775 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20775-v1.patch, HBASE-20775.patch, > HBASE-20775.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > Seems the problem is related to HBASE-20708, where we will give states other > than OPEN to meta region. > Let me dig more. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20448) update ref guide to expressly use shaded clients for examples
[ https://issues.apache.org/jira/browse/HBASE-20448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521020#comment-16521020 ] Duo Zhang commented on HBASE-20448: --- Any updates here [~busbey]? > update ref guide to expressly use shaded clients for examples > - > > Key: HBASE-20448 > URL: https://issues.apache.org/jira/browse/HBASE-20448 > Project: HBase > Issue Type: Sub-task > Components: Client, documentation, mapreduce >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 2.1.0 > > > the whole mapreduce section, especially, should be using the shaded version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20194) Basic Replication WebUI - Master
[ https://issues.apache.org/jira/browse/HBASE-20194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521019#comment-16521019 ] Duo Zhang commented on HBASE-20194: --- Is the failed UT related? [~tianjingyun] > Basic Replication WebUI - Master > > > Key: HBASE-20194 > URL: https://issues.apache.org/jira/browse/HBASE-20194 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20194.master.001.patch, > HBASE-20194.master.002.patch, HBASE-20194.master.003.patch, > HBASE-20194.master.004.patch, HBASE-20194.master.005.patch, > HBASE-20194.master.005.patch > > > subtask of HBASE-15809. Implementation of Replication WebUI on Master webpage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521018#comment-16521018 ] Duo Zhang commented on HBASE-20193: --- Ping [~tianjingyun] please update the patch? > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch, HBASE-20193.master.004.patch, > HBASE-20193.master.005.patch, HBASE-20193.master.006.patch, > HBASE-20193.master.006.patch > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20752) Make sure the regions are truly reopened after ReopenTableRegionsProcedure
[ https://issues.apache.org/jira/browse/HBASE-20752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20752: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.2 Status: Resolved (was: Patch Available) Pushed to branch-2.0. Thanks [~stack] for reviewing. > Make sure the regions are truly reopened after ReopenTableRegionsProcedure > -- > > Key: HBASE-20752 > URL: https://issues.apache.org/jira/browse/HBASE-20752 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: HBASE-20752-branch-2.0.patch, > HBASE-20752-branch-2.0.patch, HBASE-20752-branch-2.0.patch, > HBASE-20752-v1.patch, HBASE-20752.patch > > > The MRP may fail due to some reason, so we need to add some checks in > ReopenTableRegionsProcedure, if we fail to reopen a region, then we should > schedule a new MRP for it again. > And a future improvement maybe that, we can introduce a new type of procedure > to reopen a region, where we just need a simple call to RS, and do not need > to unassign it and assign it again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20575) Fail to config COMPACTION_ENABLED by hbase shell
[
https://issues.apache.org/jira/browse/HBASE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520984#comment-16520984
]
Hadoop QA commented on HBASE-20575:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} branch-1.2 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} branch-1.2 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
9s{color} | {color:red} The patch generated 12 new + 996 unchanged - 4 fixed =
1008 total (was 1000) {color} |
| {color:orange}-0{color} | {color:orange} ruby-lint {color} | {color:orange}
0m 14s{color} | {color:orange} The patch generated 19 new + 877 unchanged - 0
fixed = 896 total (was 877) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
36s{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:13ca256 |
| JIRA Issue | HBASE-20575 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928855/HBASE-20575-branch-1.2.v3.patch
|
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux 1b8d62e55ff5 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-1.2 / 20772f1 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_181 |
| Multi-JDK versions | /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_172
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_181 |
| rubocop | v0.57.2 |
| rubocop |
https://builds.apache.org/job/PreCommit-HBASE-Build/13356/artifact/patchprocess/diff-patch-rubocop.txt
|
| ruby-lint | v2.3.1 |
| ruby-lint |
https://builds.apache.org/job/PreCommit-HBASE-Build/13356/artifact/patchprocess/diff-patch-ruby-lint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13356/testReport/ |
| Max. process+thread count | 619 (vs. ulimit of 1) |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13356/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Fail to config COMPACTION_ENABLED by hbase shell
>
>
> Key: HBASE-20575
> URL: https://issues.apache.org/jira/browse/HBASE-20575
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.3.2
>Reporter: Chia-Ping Tsai
>Assignee: Mingdao Yang
>Priority: Major
> Fix For: 1.2.7, 1.3.3
>
> Attachments: HBASE-20575-branch-1.2.patch,
> HBASE-20575-branch-1.2.v1.patch, HBASE-20575-branch-1.2.v2.patch,
> HBASE-20575-branch-1.2.v3.patch
>
>
> HBASE-19340
