[jira] [Updated] (HBASE-17746) TestSimpleRpcScheduler.testCoDelScheduling is broken
[ https://issues.apache.org/jira/browse/HBASE-17746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-17746: --- Attachment: HBASE-17746.branch-1.001.patch > TestSimpleRpcScheduler.testCoDelScheduling is broken > > > Key: HBASE-17746 > URL: https://issues.apache.org/jira/browse/HBASE-17746 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0 > Environment: OS: Ubuntu 14.04 > Arch: x86_64 > Linux x 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 > x86_64 x86_64 x86_64 GNU/Linux >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17746.branch-1.001.patch, HBASE-17746.v0.patch > > > Command: > export JAVA_TOOL_OPTIONS="-Xmx8G -Xms2G" > mvn clean install -X > Output: > Results : > Failed tests: > TestSimpleRpcScheduler.testCoDelScheduling:469 None of these calls should > have been discarded expected:<0> but was:<15> > Tests run: 1191, Failures: 1, Errors: 0, Skipped: 12 > Output of "TestSimpleRpcScheduler.txt" > --- > Test set: org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > --- > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.642 sec <<< > FAILURE! - in org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > testCoDelScheduling(org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler) Time > elapsed: 0.563 sec <<< FAILURE! > java.lang.AssertionError: None of these calls should have been discarded > expected:<0> but was:<15> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testCoDelScheduling(TestSimpleRpcScheduler.java:469) > -- > Output of last few lines of "TestSimpleRpcScheduler-output.txt" > 2017-03-07 16:09:16,580 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testCoDelScheduling Thread=5 (was 5), > OpenFileDescriptor=250 (was 250), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 157) - ProcessCount > LEAK? -, AvailableMemoryMB=1734 (was 1762) > 2017-03-07 16:09:16,604 INFO [main] hbase.ResourceChecker(148): before: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5, > OpenFileDescriptor=252, MaxFileDescriptor=4096, SystemLoadAverage=607, > ProcessCount=158, AvailableMemoryMB=1734 > 2017-03-07 16:09:16,722 INFO [main] ipc.RpcExecutor(145): RpcExecutor name > using fifo as call queue; numCallQueues=1; maxQueueLength=5; handlerCount=0 > 2017-03-07 16:09:16,722 DEBUG [main] ipc.RpcExecutor(215): Started > RpcServer.deafult.FPBQ.Fifo.handler=0,queue=0,port=0 > 2017-03-07 16:09:16,839 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5 (was 5), > OpenFileDescriptor=250 (was 252), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 158), > AvailableMemoryMB=1703 (was 1734) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17763) IPCUtil.wrapException will wrap DoNotRetryIOException with IOException
[ https://issues.apache.org/jira/browse/HBASE-17763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-17763: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~saint@gmail.com] for reviewing. > IPCUtil.wrapException will wrap DoNotRetryIOException with IOException > -- > > Key: HBASE-17763 > URL: https://issues.apache.org/jira/browse/HBASE-17763 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17763.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904620#comment-15904620 ] Anoop Sam John commented on HBASE-17765: So here the merge means the cell data also will get copied to new MSLAB right? Or it is just merge of the index data? Ya make that 30 to be configurable is fine IMO. In ur tests how much this #segments in pipeline gets to?As per the math it can not be more than 16 no? Default 128 MB flush size and 4x as blocking factor. And 25% of flush size is what the in memory flush cap.So we can have 4 * 4 segments at max in pipeline? Or my math is somewhere wrong? Accordingly the default of this config we need to adjust. IMO than a 30 value, we should do this way.. When the config values of this in memory flush factor and/or blocking factor changes, we need to adjust this new config so as to make the default way of no merge. Ya one can always change it then. > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17746) TestSimpleRpcScheduler.testCoDelScheduling is broken
[ https://issues.apache.org/jira/browse/HBASE-17746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904612#comment-15904612 ] Guanghao Zhang commented on HBASE-17746: For FifoQueueType, it should use FastPathBalancedQueueRpcExecutor, too. > TestSimpleRpcScheduler.testCoDelScheduling is broken > > > Key: HBASE-17746 > URL: https://issues.apache.org/jira/browse/HBASE-17746 > Project: HBase > Issue Type: Bug > Components: integration tests >Affects Versions: 2.0.0 > Environment: OS: Ubuntu 14.04 > Arch: x86_64 > Linux x 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 > x86_64 x86_64 x86_64 GNU/Linux >Reporter: Anup Halarnkar >Assignee: Chia-Ping Tsai > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17746.v0.patch > > > Command: > export JAVA_TOOL_OPTIONS="-Xmx8G -Xms2G" > mvn clean install -X > Output: > Results : > Failed tests: > TestSimpleRpcScheduler.testCoDelScheduling:469 None of these calls should > have been discarded expected:<0> but was:<15> > Tests run: 1191, Failures: 1, Errors: 0, Skipped: 12 > Output of "TestSimpleRpcScheduler.txt" > --- > Test set: org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > --- > Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.642 sec <<< > FAILURE! - in org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler > testCoDelScheduling(org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler) Time > elapsed: 0.563 sec <<< FAILURE! > java.lang.AssertionError: None of these calls should have been discarded > expected:<0> but was:<15> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at > org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler.testCoDelScheduling(TestSimpleRpcScheduler.java:469) > -- > Output of last few lines of "TestSimpleRpcScheduler-output.txt" > 2017-03-07 16:09:16,580 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testCoDelScheduling Thread=5 (was 5), > OpenFileDescriptor=250 (was 250), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 157) - ProcessCount > LEAK? -, AvailableMemoryMB=1734 (was 1762) > 2017-03-07 16:09:16,604 INFO [main] hbase.ResourceChecker(148): before: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5, > OpenFileDescriptor=252, MaxFileDescriptor=4096, SystemLoadAverage=607, > ProcessCount=158, AvailableMemoryMB=1734 > 2017-03-07 16:09:16,722 INFO [main] ipc.RpcExecutor(145): RpcExecutor name > using fifo as call queue; numCallQueues=1; maxQueueLength=5; handlerCount=0 > 2017-03-07 16:09:16,722 DEBUG [main] ipc.RpcExecutor(215): Started > RpcServer.deafult.FPBQ.Fifo.handler=0,queue=0,port=0 > 2017-03-07 16:09:16,839 INFO [main] hbase.ResourceChecker(172): after: > ipc.TestSimpleRpcScheduler#testSoftAndHardQueueLimits Thread=5 (was 5), > OpenFileDescriptor=250 (was 252), MaxFileDescriptor=4096 (was 4096), > SystemLoadAverage=607 (was 607), ProcessCount=158 (was 158), > AvailableMemoryMB=1703 (was 1734) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17765: --- Fix Version/s: 2.0.0 > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16584: -- Attachment: HBASE-16584-branch-1-v3.patch > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, > HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch, > HBASE-16584-branch-1-v3.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (HBASE-17767) hbase CMS gc pause program
[ https://issues.apache.org/jira/browse/HBASE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-17767. --- Resolution: Invalid > hbase CMS gc pause program > > > Key: HBASE-17767 > URL: https://issues.apache.org/jira/browse/HBASE-17767 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: gehaijiang > > cms gc pause serious program: > 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: > 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] > 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, > real=2.05 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: > 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start] > 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 > secs] [Times: user=1.23 sys=0.08, real=0.22 secs] > 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: > 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] > 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, > real=2.41 secs] > 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: > 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: > user=0.01 sys=0.00, real=0.01 secs] > 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 > secs] [Times: user=5.72 sys=0.05, real=0.43 secs] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start] > 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: > 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] > 2017-03-10T10:15:36.291+0800: 4555926.928: > [CMS-concurrent-abortable-preclean-start] > 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: > 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] > 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, > real=2.62 secs] > 2017-03-10T10:15:41.012+0800: 4555931.648: > [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 > sys=0.87, real=4.72 secs] > 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K > (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: > [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: > 2011637K->340736K(3067136K), 2.0773980 secs] > 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, > real=2.07 secs] > regionserver JVM config: > export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m > -XX:MaxPermSize=256m -Xms96G -Xmx96G" > export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 > -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled > -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -verbose:gc > -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 > -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16488) Starting namespace and quota services in master startup asynchronizely
[
https://issues.apache.org/jira/browse/HBASE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904487#comment-15904487
]
Hadoop QA commented on HBASE-16488:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 24s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 24s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 45s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 112m 21s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestCompactionInDeadRegionServer |
| | hadoop.hbase.quotas.TestQuotaAdmin |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.master.TestMasterBalanceThrottling |
| | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
| Timed out junit tests |
org.apache.hadoop.hbase.util.TestMiniClusterLoadEncoded |
| | org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| |
[jira] [Updated] (HBASE-17338) Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB
[ https://issues.apache.org/jira/browse/HBASE-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-17338: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the reviews > Treat Cell data size under global memstore heap size only when that Cell can > not be copied to MSLAB > --- > > Key: HBASE-17338 > URL: https://issues.apache.org/jira/browse/HBASE-17338 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-17338.patch, HBASE-17338_V2.patch, > HBASE-17338_V2.patch, HBASE-17338_V4.patch, HBASE-17338_V5.patch, > HBASE-17338_V6.patch > > > We have only data size and heap overhead being tracked globally. Off heap > memstore works with off heap backed MSLAB pool. But a cell, when added to > memstore, not always getting copied to MSLAB. Append/Increment ops doing an > upsert, dont use MSLAB. Also based on the Cell size, we sometimes avoid > MSLAB copy. But now we track these cell data size also under the global > memstore data size which indicated off heap size in case of off heap > memstore. For global checks for flushes (against lower/upper watermark > levels), we check this size against max off heap memstore size. We do check > heap overhead against global heap memstore size (Defaults to 40% of xmx) But > for such cells the data size also should be accounted under the heap overhead. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904485#comment-15904485
]
Guangxu Cheng commented on HBASE-17758:
---
v2 takes the suggestion from [~yuzhih...@gmail.com]
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904483#comment-15904483 ] Anoop Sam John commented on HBASE-16417: bq.how you think the numbers would change if 'default' 1k values were used? Would we see the same benefit do you think? I guess the improvement will be on decreasing side as the value size increases. When val size is low like 100 bytes, there will be more #cells added to CLSM before a flush to disk and more overhead in heap size. The in memory flush (BASIC) reduces this overhead greatly and also avoids the situation of adding more and more cells to an already large CSLM. But when val size is more, we might be dealing with much lesser #cells. This is what my thinking is. bq.The current default active segment size cap for in-memory flush is 1/4 the memstore size cap for disk flush. Which means that the expected number of segments in the pipeline is 4/2=2. Why 4/2? Sorry am not getting.. As 25% is the in memory flush cap, ideally 4 segments can be there (It is basic type and no merge).. But we have a blocking factor of 4 by default for the memstore flush size. Means we will allow max memstore size of 128 MB * 4 before a forceful flush to disk. So 4 * 4 = 16 can be the max #segments? My math is wrong some where? I was not getting that /2 thing. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17758) [RSGROUP] Add shell command to move servers and tables at the same time
[
https://issues.apache.org/jira/browse/HBASE-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangxu Cheng updated HBASE-17758:
--
Attachment: HBASE-17758-v2.patch
> [RSGROUP] Add shell command to move servers and tables at the same time
> ---
>
> Key: HBASE-17758
> URL: https://issues.apache.org/jira/browse/HBASE-17758
> Project: HBase
> Issue Type: New Feature
> Components: rsgroup
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
> Attachments: HBASE-17758-v1.patch, HBASE-17758-v2.patch
>
>
> Currently add a new group perform the following steps:
> {code:javascript}
> hbase(main):001:0> add_rsgroup 'mygroup'
> Took 0.3840 seconds
> hbase(main):002:0> move_servers_rsgroup
> 'mygroup',['hbase-rs-01:16030','hbase-rs-02:16030']
> Took 3.5040 seconds
> hbase(main):003:0> move_tables_rsgroup 'mygroup',['example']
> Took 0.2710 seconds
> {code}
> 1. move_servers_rsgroup will unassign all regions on hbase-rs-01 and
> hbase-rs-02
> 2. move_tables_rsgroup will also reassign all the regions of the table
> example.
> This will lead to a large number of regions to migrate and affect the data
> locality.
> However, some regions of the table that are distributed on hbase-rs-01 and
> hbase-rs-02, do not need to be moved.
> So,we need a new shell command *move_servers_tables_rsgroup* which minimizes
> the number of regions needed to move.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17767) hbase CMS gc pause program
[ https://issues.apache.org/jira/browse/HBASE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904463#comment-15904463 ] gehaijiang commented on HBASE-17767: thanks @ Apache9 Duo Zhang > hbase CMS gc pause program > > > Key: HBASE-17767 > URL: https://issues.apache.org/jira/browse/HBASE-17767 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: gehaijiang > > cms gc pause serious program: > 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: > 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] > 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, > real=2.05 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: > 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start] > 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 > secs] [Times: user=1.23 sys=0.08, real=0.22 secs] > 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: > 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] > 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, > real=2.41 secs] > 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: > 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: > user=0.01 sys=0.00, real=0.01 secs] > 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 > secs] [Times: user=5.72 sys=0.05, real=0.43 secs] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start] > 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: > 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] > 2017-03-10T10:15:36.291+0800: 4555926.928: > [CMS-concurrent-abortable-preclean-start] > 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: > 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] > 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, > real=2.62 secs] > 2017-03-10T10:15:41.012+0800: 4555931.648: > [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 > sys=0.87, real=4.72 secs] > 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K > (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: > [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: > 2011637K->340736K(3067136K), 2.0773980 secs] > 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, > real=2.07 secs] > regionserver JVM config: > export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m > -XX:MaxPermSize=256m -Xms96G -Xmx96G" > export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 > -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled > -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -verbose:gc > -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 > -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17767) hbase CMS gc pause program
[ https://issues.apache.org/jira/browse/HBASE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904451#comment-15904451 ] Duo Zhang commented on HBASE-17767: --- [~gehaijiang] Have you found some problems that need to be fixed? Or do you only want to ask why CMS hangs for a long time? If the latter, I suggest you sending an email to the mailing-list. u...@hbase.apache.org Thanks. > hbase CMS gc pause program > > > Key: HBASE-17767 > URL: https://issues.apache.org/jira/browse/HBASE-17767 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: gehaijiang > > cms gc pause serious program: > 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: > 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] > 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, > real=2.05 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: > 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs] > 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start] > 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 > secs] [Times: user=1.23 sys=0.08, real=0.22 secs] > 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: > 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] > 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, > real=2.41 secs] > 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: > 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: > user=0.01 sys=0.00, real=0.01 secs] > 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 > secs] [Times: user=5.72 sys=0.05, real=0.43 secs] > 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start] > 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: > 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] > 2017-03-10T10:15:36.291+0800: 4555926.928: > [CMS-concurrent-abortable-preclean-start] > 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: > 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] > 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, > real=2.62 secs] > 2017-03-10T10:15:41.012+0800: 4555931.648: > [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 > sys=0.87, real=4.72 secs] > 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K > (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: > [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: > 2011637K->340736K(3067136K), 2.0773980 secs] > 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, > real=2.07 secs] > regionserver JVM config: > export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m > -XX:MaxPermSize=256m -Xms96G -Xmx96G" > export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 > -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled > -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails > -XX:+PrintGCDateStamps -verbose:gc > -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 > -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[ https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904442#comment-15904442 ] Hudson commented on HBASE-16977: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #123 (See [https://builds.apache.org/job/HBase-1.3-JDK7/123/]) HBASE-16977 VerifyReplication should log a printable representation of (garyh: rev 729239c8d13bcf15b8f87030a6f23b723d6f7227) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java > VerifyReplication should log a printable representation of the row keys > --- > > Key: HBASE-16977 > URL: https://issues.apache.org/jira/browse/HBASE-16977 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16977.branch-1.3.001.patch, > HBASE-16977.master.001.patch, HBASE-16977.V1.patch > > > VerifyReplication prints out the row keys for offending rows in the task logs > for the MR job. However, the log is useless if the row key contains non > printable characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16302) age of last shipped op and age of last applied op should be histograms
[ https://issues.apache.org/jira/browse/HBASE-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904441#comment-15904441 ] Hudson commented on HBASE-16302: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #123 (See [https://builds.apache.org/job/HBase-1.3-JDK7/123/]) HBASE-17579 Backport HBASE-16302 to 1.3.1 (garyh: rev d74b7270852289c2d05f139b1142f034f9348897) * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableHistogram.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java > age of last shipped op and age of last applied op should be histograms > -- > > Key: HBASE-16302 > URL: https://issues.apache.org/jira/browse/HBASE-16302 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16302.patch.v0.patch > > > Replication exports metric ageOfLastShippedOp as an indication of how much > replication is lagging. But, with multiwal enabled, it's not representative > because replication could be lagging for a long time for one wal group > (something wrong with a particular region) while being fine for others. The > ageOfLastShippedOp becomes a useless metric for alerting in such a case. > Also, since there is no mapping between individual replication sources and > replication sinks, the age of last applied op can be a highly spiky metric if > only certain replication sources are lagging. > We should use histograms for these metrics and use maximum value of this > histogram to report replication lag when building stats. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904439#comment-15904439 ] Hudson commented on HBASE-17579: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #123 (See [https://builds.apache.org/job/HBase-1.3-JDK7/123/]) HBASE-17579 Backport HBASE-16302 to 1.3.1 (garyh: rev d74b7270852289c2d05f139b1142f034f9348897) * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableHistogram.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java > Backport HBASE-16302 to 1.3.1 > - > > Key: HBASE-17579 > URL: https://issues.apache.org/jira/browse/HBASE-17579 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.1 > > Attachments: HBASE-17579.branch-1.3.001.patch, > HBASE-17579.branch-1.3.002.patch, HBASE-17579.branch-1.3.003.patch > > > This is a simple enough change to be included in 1.3.1, and replication > monitoring essentially breaks without this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17767) hbase CMS gc pause program
gehaijiang created HBASE-17767: -- Summary: hbase CMS gc pause program Key: HBASE-17767 URL: https://issues.apache.org/jira/browse/HBASE-17767 Project: HBase Issue Type: Bug Affects Versions: 1.1.2 Reporter: gehaijiang cms gc pause serious program: 2017-03-10T10:15:29.524+0800: 4555920.160: [GC2017-03-10T10:15:29.524+0800: 4555920.160: [ParNew: 3067133K->340736K(3067136K), 2.0586980 secs] 80328431K->78058138K(100322560K), 2.0590280 secs] [Times: user=3.94 sys=0.34, real=2.05 secs] 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-sweep: 1441.773/1618.869 secs] [Times: user=2518.60 sys=59.25, real=1618.62 secs] 2017-03-10T10:15:32.911+0800: 4555923.547: [CMS-concurrent-reset-start] 2017-03-10T10:15:33.126+0800: 4555923.762: [CMS-concurrent-reset: 0.215/0.215 secs] [Times: user=1.23 sys=0.08, real=0.22 secs] 2017-03-10T10:15:33.236+0800: 4555923.873: [GC2017-03-10T10:15:33.237+0800: 4555923.873: [ParNew: 3067011K->340736K(3067136K), 2.4140270 secs] 80615855K->78315999K(100322560K), 2.4144230 secs] [Times: user=4.63 sys=0.36, real=2.41 secs] 2017-03-10T10:15:35.655+0800: 4555926.292: [GC [1 CMS-initial-mark: 77975263K(97255424K)] 78316286K(100322560K), 0.0149650 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 2017-03-10T10:15:35.671+0800: 4555926.307: [CMS-concurrent-mark-start] 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-mark: 0.427/0.427 secs] [Times: user=5.72 sys=0.05, real=0.43 secs] 2017-03-10T10:15:36.098+0800: 4555926.734: [CMS-concurrent-preclean-start] 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-preclean: 0.192/0.193 secs] [Times: user=0.80 sys=0.03, real=0.19 secs] 2017-03-10T10:15:36.291+0800: 4555926.928: [CMS-concurrent-abortable-preclean-start] 2017-03-10T10:15:37.378+0800: 4555928.014: [GC2017-03-10T10:15:37.378+0800: 4555928.014: [ParNew: 3067083K->340736K(3067136K), 2.6221190 secs] 81042347K->78771078K(100322560K), 2.6224970 secs] [Times: user=4.79 sys=0.48, real=2.62 secs] 2017-03-10T10:15:41.012+0800: 4555931.648: [CMS-concurrent-abortable-preclean: 2.083/4.721 secs] [Times: user=13.51 sys=0.87, real=4.72 secs] 2017-03-10T10:15:41.015+0800: 4555931.652: [GC[YG occupancy: 2011637 K (3067136 K)]2017-03-10T10:15:41.016+0800: 4555931.652: [GC2017-03-10T10:15:41.016+0800: 4555931.652: [ParNew: 2011637K->340736K(3067136K), 2.0773980 secs] 80441979K->79117650K(100322560K), 2.0777380 secs] [Times: user=4.09 sys=0.38, real=2.07 secs] regionserver JVM config: export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=256m -XX:MaxPermSize=256m -Xms96G -Xmx96G" export HBASE_OPTS="$HBASE_OPTS -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+CMSConcurrentMTEnabled -XX:ParallelGCThreads=40 -XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -verbose:gc -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSScavengeBeforeRemark -XX:+HeapDumpOnOutOfMemoryError -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17745) Support short circuit connection for master services
[ https://issues.apache.org/jira/browse/HBASE-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904428#comment-15904428 ] Hudson commented on HBASE-17745: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2644 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2644/]) HBASE-17745 Support short circuit connection for master services (liyu: rev c63a871176cc55d14ac2047091b48743a8cce782) * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java > Support short circuit connection for master services > > > Key: HBASE-17745 > URL: https://issues.apache.org/jira/browse/HBASE-17745 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0 > > Attachments: HBASE-17745.patch, HBASE-17745.v2.patch, > HBASE-17745.v2.trival.patch, HBASE-17745.v2.trival.patch, HBASE-17745.v3.patch > > > As titled, now we have short circuit connection, but no short circuit for > master services, and we propose to support it in this JIRA. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17731) Fractional latency reporting in MultiThreadedAction
[ https://issues.apache.org/jira/browse/HBASE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904426#comment-15904426 ] Hudson commented on HBASE-17731: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2644 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2644/]) HBASE-17731 Fractional latency reporting in MultiThreadedAction (apurtell: rev 7da0feea8dd4236b520bf48b2a84c52bf2910e56) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java > Fractional latency reporting in MultiThreadedAction > --- > > Key: HBASE-17731 > URL: https://issues.apache.org/jira/browse/HBASE-17731 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17731.patch > > > When average latency is less than one millisecond the LoadTestTool tool > reports a latency of 0. Better to report a fraction out to a couple of > decimal points. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17736) Some options can't be configured by the shell
[ https://issues.apache.org/jira/browse/HBASE-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904425#comment-15904425 ] Hudson commented on HBASE-17736: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2644 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2644/]) HBASE-17736 Some options can't be configured by the shell (apurtell: rev 26928ff912643c19167646ec22823518be186f0d) * (edit) hbase-shell/src/test/ruby/hbase/admin_test.rb * (edit) hbase-shell/src/main/ruby/hbase/admin.rb > Some options can't be configured by the shell > - > > Key: HBASE-17736 > URL: https://issues.apache.org/jira/browse/HBASE-17736 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17736.branch-1.v0.patch, HBASE-17736.v0.patch, > HBASE-17736.v1.patch, HBASE-17736.v2.patch > > > The lost options for table are shown below. > # setFlushPolicyClassName > # setPriority > # setRegionMemstoreReplication > # setRegionSplitPolicyClassName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[ https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904424#comment-15904424 ] Hudson commented on HBASE-16977: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2644 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2644/]) HBASE-16977 VerifyReplication should log a printable representation of (garyh: rev d4cbff58cc67577ff66507b5d2bc0c7150bbd21b) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java > VerifyReplication should log a printable representation of the row keys > --- > > Key: HBASE-16977 > URL: https://issues.apache.org/jira/browse/HBASE-16977 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16977.branch-1.3.001.patch, > HBASE-16977.master.001.patch, HBASE-16977.V1.patch > > > VerifyReplication prints out the row keys for offending rows in the task logs > for the MR job. However, the log is useless if the row key contains non > printable characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Attachment: HBASE-17125.master.002.patch
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904378#comment-15904378 ] Anoop Sam John commented on HBASE-17138: Not as such.. The effort for backport seems more.. Ya Sunyu did all the backport to their custom version but on branch-1 it wont be a direct fit.. And then effort of testing.. Seems [~carp84] was not having that much bandwidth.. Is there some interest for collaboration guys? Let us know.. We can do any help as needed. > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge
[jira] [Commented] (HBASE-15429) Add a split policy for busy regions
[ https://issues.apache.org/jira/browse/HBASE-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904325#comment-15904325 ] Hudson commented on HBASE-15429: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #6 (See [https://builds.apache.org/job/HBase-1.3-IT/6/]) HBASE-15429 Add split policy for busy regions (garyh: rev 0db1e0f7a80da125d3819e5e6efd8b7af09ffe31) * (edit) hbase-common/src/main/resources/hbase-default.xml * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BusyRegionSplitPolicy.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java > Add a split policy for busy regions > --- > > Key: HBASE-15429 > URL: https://issues.apache.org/jira/browse/HBASE-15429 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-15429.branch-1.patch, HBASE-15429.patch, > HBASE-15429-V1.patch, HBASE-15429-V2.patch > > > Currently, all region split policies are based on size. However, in certain > cases, it is a wise choice to make a split decision based on number of > requests to the region and split busy regions. > A crude metric is that if a region blocks writes often and throws > RegionTooBusyExceoption, it's probably a good idea to split it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16302) age of last shipped op and age of last applied op should be histograms
[ https://issues.apache.org/jira/browse/HBASE-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904323#comment-15904323 ] Hudson commented on HBASE-16302: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #6 (See [https://builds.apache.org/job/HBase-1.3-IT/6/]) HBASE-17579 Backport HBASE-16302 to 1.3.1 (garyh: rev d74b7270852289c2d05f139b1142f034f9348897) * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableHistogram.java > age of last shipped op and age of last applied op should be histograms > -- > > Key: HBASE-16302 > URL: https://issues.apache.org/jira/browse/HBASE-16302 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16302.patch.v0.patch > > > Replication exports metric ageOfLastShippedOp as an indication of how much > replication is lagging. But, with multiwal enabled, it's not representative > because replication could be lagging for a long time for one wal group > (something wrong with a particular region) while being fine for others. The > ageOfLastShippedOp becomes a useless metric for alerting in such a case. > Also, since there is no mapping between individual replication sources and > replication sinks, the age of last applied op can be a highly spiky metric if > only certain replication sources are lagging. > We should use histograms for these metrics and use maximum value of this > histogram to report replication lag when building stats. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904321#comment-15904321 ] Hudson commented on HBASE-17579: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #6 (See [https://builds.apache.org/job/HBase-1.3-IT/6/]) HBASE-17579 Backport HBASE-16302 to 1.3.1 (garyh: rev d74b7270852289c2d05f139b1142f034f9348897) * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/MutableHistogram.java > Backport HBASE-16302 to 1.3.1 > - > > Key: HBASE-17579 > URL: https://issues.apache.org/jira/browse/HBASE-17579 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.1 > > Attachments: HBASE-17579.branch-1.3.001.patch, > HBASE-17579.branch-1.3.002.patch, HBASE-17579.branch-1.3.003.patch > > > This is a simple enough change to be included in 1.3.1, and replication > monitoring essentially breaks without this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[ https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904324#comment-15904324 ] Hudson commented on HBASE-16977: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #6 (See [https://builds.apache.org/job/HBase-1.3-IT/6/]) HBASE-16977 VerifyReplication should log a printable representation of (garyh: rev 729239c8d13bcf15b8f87030a6f23b723d6f7227) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java > VerifyReplication should log a printable representation of the row keys > --- > > Key: HBASE-16977 > URL: https://issues.apache.org/jira/browse/HBASE-16977 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16977.branch-1.3.001.patch, > HBASE-16977.master.001.patch, HBASE-16977.V1.patch > > > VerifyReplication prints out the row keys for offending rows in the task logs > for the MR job. However, the log is useless if the row key contains non > printable characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17761) Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of race condition
[ https://issues.apache.org/jira/browse/HBASE-17761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904320#comment-15904320 ] Hudson commented on HBASE-17761: FAILURE: Integrated in Jenkins build HBase-1.4 #663 (See [https://builds.apache.org/job/HBase-1.4/663/]) HBASE-17761: Test TestRemoveRegionMetrics.testMoveRegion fails (esteban: rev 3d9520b1403346cbe7a6322cf6c2632197d79f7a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRemoveRegionMetrics.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java > Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of > race condition > -- > > Key: HBASE-17761 > URL: https://issues.apache.org/jira/browse/HBASE-17761 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17761_branch-1.v1.patch, > HBASE-17761-branch-1.v1.patch, HBASE-17761.patch, HBASE-17761.v1.patch > > > After moving the region code waits till all regions are assigned but not for > region to go online on a destination server. On branch-1 TestAdmin1.java has > a function moveRegionAndWait() that moves the region and waits for the region > to become online on destination server. We need to use that function in this > test. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16488) Starting namespace and quota services in master startup asynchronizely
[ https://issues.apache.org/jira/browse/HBASE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16488: --- Attachment: HBASE-16488.v5-branch-1.patch > Starting namespace and quota services in master startup asynchronizely > -- > > Key: HBASE-16488 > URL: https://issues.apache.org/jira/browse/HBASE-16488 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Attachments: HBASE-16488.v1-branch-1.patch, > HBASE-16488.v1-master.patch, HBASE-16488.v2-branch-1.patch, > HBASE-16488.v2-branch-1.patch, HBASE-16488.v3-branch-1.patch, > HBASE-16488.v3-branch-1.patch, HBASE-16488.v4-branch-1.patch, > HBASE-16488.v5-branch-1.patch > > > From time to time, during internal IT test and from customer, we often see > master initialization failed due to namespace table region takes long time to > assign (eg. sometimes split log takes long time or hanging; or sometimes RS > is temporarily not available; sometimes due to some unknown assignment > issue). In the past, there was some proposal to improve this situation, eg. > HBASE-13556 / HBASE-14190 (Assign system tables ahead of user region > assignment) or HBASE-13557 (Special WAL handling for system tables) or > HBASE-14623 (Implement dedicated WAL for system tables). > This JIRA proposes another way to solve this master initialization fail > issue: namespace service is only used by a handful operations (eg. create > table / namespace DDL / get namespace API / some RS group DDL). Only quota > manager depends on it and quota management is off by default. Therefore, > namespace service is not really needed for master to be functional. So we > could start namespace service asynchronizely without blocking master startup. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17707) New More Accurate Table Skew cost function/generator
[ https://issues.apache.org/jira/browse/HBASE-17707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904297#comment-15904297 ] Ted Yu commented on HBASE-17707: [~kahliloppenheimer]: How about using the new table skew cost function when there is no region replica in the cluster and using the existing one when there is region replica ? > New More Accurate Table Skew cost function/generator > > > Key: HBASE-17707 > URL: https://issues.apache.org/jira/browse/HBASE-17707 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 1.2.0 > Environment: CentOS Derivative with a derivative of the 3.18.43 > kernel. HBase on CDH5.9.0 with some patches. HDFS CDH 5.9.0 with no patches. >Reporter: Kahlil Oppenheimer >Assignee: Kahlil Oppenheimer >Priority: Minor > Fix For: 2.0 > > Attachments: HBASE-17707-00.patch, HBASE-17707-01.patch, > HBASE-17707-02.patch, HBASE-17707-03.patch, HBASE-17707-04.patch, > HBASE-17707-05.patch, HBASE-17707-06.patch, HBASE-17707-07.patch, > HBASE-17707-08.patch, test-balancer2-13617.out > > > This patch includes new version of the TableSkewCostFunction and a new > TableSkewCandidateGenerator. > The new TableSkewCostFunction computes table skew by counting the minimal > number of region moves required for a given table to perfectly balance the > table across the cluster (i.e. as if the regions from that table had been > round-robin-ed across the cluster). This number of moves is computer for each > table, then normalized to a score between 0-1 by dividing by the number of > moves required in the absolute worst case (i.e. the entire table is stored on > one server), and stored in an array. The cost function then takes a weighted > average of the average and maximum value across all tables. The weights in > this average are configurable to allow for certain users to more strongly > penalize situations where one table is skewed versus where every table is a > little bit skewed. To better spread this value more evenly across the range > 0-1, we take the square root of the weighted average to get the final value. > The new TableSkewCandidateGenerator generates region moves/swaps to optimize > the above TableSkewCostFunction. It first simply tries to move regions until > each server has the right number of regions, then it swaps regions around > such that each region swap improves table skew across the cluster. > We tested the cost function and generator in our production clusters with > 100s of TBs of data and 100s of tables across dozens of servers and found > both to be very performant and accurate. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904295#comment-15904295 ] Enis Soztutar commented on HBASE-14070: --- Yep, this would be pretty good if we can get it in. Last time I've looked at it is some time ago. If you want to pick it up, go for it. > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17754) [C++] RawAsyncTable
[
https://issues.apache.org/jira/browse/HBASE-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-17754:
--
Attachment: hbase-17754_v1.patch
v1. The tests are working now. I've got rid of bunch of complexity in the
templates, and got rid of the retry-test in favor of client-test. Some debug
logging statements maybe lying around.
Xiaobing please take a look whenever you find time.
> [C++] RawAsyncTable
> ---
>
> Key: HBASE-17754
> URL: https://issues.apache.org/jira/browse/HBASE-17754
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17754_v0.patch, hbase-17754_v1.patch
>
>
> We need RawAsyncTable to connect {{table.h}} to the async retrying callers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17764) Solve TestMultiSlaveReplication flakiness
[ https://issues.apache.org/jira/browse/HBASE-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-17764: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) > Solve TestMultiSlaveReplication flakiness > -- > > Key: HBASE-17764 > URL: https://issues.apache.org/jira/browse/HBASE-17764 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Fix For: 1.4.0 > > Attachments: HBASE-17764.v1-branch-1.patch > > > When I doing testing for a feature I am working on, the > TestMultiSlaveReplication test was flaky. > TestMultiSlaveReplication#testMultiSlaveReplication would always pass when > running alone; however, it would fail occasionally when running with the > suite (the other test TestMultiSlaveReplication#testZKLockCleaner always run > first and I suspect that it withhold some zookeeper resource, which caused > the problem). > After I set start and shutdown zk cluster before and after each test, I no > longer see the problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904286#comment-15904286
]
Jerry He commented on HBASE-15597:
--
v3 LGTM
{code}
val QUERY_CACHESIZE = "hbase.spark.query.cachesize"
val QUERY_BATCHSIZE = "hbase.spark.query.batchsize"
{code}
{code}
val TIMERANGE_START = "hbase.spark.query.timerange.start"
val TIMERANGE_END = "hbase.spark.query.timerange.end"
val MAX_VERSIONS = "hbase.spark.query.maxVersions"
val MAX_VERSIONS = "hbase.spark.query.maxVersions"
{code}
It is good we use 'query' instead of 'scan' in the context of Spark query.
'cachesize' is named 'cachedrows' in TableInputFormat. Let's follow it?
{code}
/** The number of rows for caching that will be passed to scanners. */
public static final String SCAN_CACHEDROWS =
"hbase.mapreduce.scan.cachedrows";
{code}
Please add some comments on these names if you can.
> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Yi Liang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch,
> HBASE-15597-V3.patch
>
>
> This should be considered a blocker for backport to branch-1 since it will
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since
> our configurations keys for the spark integration all relate to that system,
> the prefix for all configuration keys (excluding those cases where we need to
> do something special due to restrictions in how properties are handled by
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we
> should review all of our configuration keys to make sure they either conform
> to the "hbase.spark" prefix or they have a comment documenting why they need
> to be otherwise.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-11033) Allow test-patch.sh to run selected test(s) if patch contains changes to test classes only
[ https://issues.apache.org/jira/browse/HBASE-11033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HBASE-11033. Resolution: Won't Fix > Allow test-patch.sh to run selected test(s) if patch contains changes to test > classes only > -- > > Key: HBASE-11033 > URL: https://issues.apache.org/jira/browse/HBASE-11033 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Priority: Minor > Labels: precommit > > A patch may touch test classes only. > In this case, test-patch.sh can detect the tests affected and run these tests > only. > Special case: if TestingUtility (HConnectionTestingUtility, > HBaseTestingUtility, etc) is changed, all tests need to be run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15429) Add a split policy for busy regions
[ https://issues.apache.org/jira/browse/HBASE-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-15429: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed the branch-1 patch to branch-1 and branch-1.3. [~mantonov] I pulled this in to 1.3 since it's completely self contained and not used unless enabled. If you see an issue, let me know and I can revert. > Add a split policy for busy regions > --- > > Key: HBASE-15429 > URL: https://issues.apache.org/jira/browse/HBASE-15429 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-15429.branch-1.patch, HBASE-15429.patch, > HBASE-15429-V1.patch, HBASE-15429-V2.patch > > > Currently, all region split policies are based on size. However, in certain > cases, it is a wise choice to make a split decision based on number of > requests to the region and split busy regions. > A crude metric is that if a region blocks writes often and throws > RegionTooBusyExceoption, it's probably a good idea to split it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17763) IPCUtil.wrapException will wrap DoNotRetryIOException with IOException
[ https://issues.apache.org/jira/browse/HBASE-17763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stack updated HBASE-17763: -- Ok > IPCUtil.wrapException will wrap DoNotRetryIOException with IOException > -- > > Key: HBASE-17763 > URL: https://issues.apache.org/jira/browse/HBASE-17763 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17763.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16584) Backport the new ipc implementation in HBASE-16432 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904217#comment-15904217 ] Duo Zhang commented on HBASE-16584: --- Good. Only two failed UTs. Let me check. > Backport the new ipc implementation in HBASE-16432 to branch-1 > -- > > Key: HBASE-16584 > URL: https://issues.apache.org/jira/browse/HBASE-16584 > Project: HBase > Issue Type: Task > Components: Client, IPC/RPC >Affects Versions: 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16584-branch-1.patch, > HBASE-16584-branch-1-v1.patch, HBASE-16584-branch-1-v2.patch > > > Two problems. > First, as RpcCliemtImpl is the default implementation on branch-1, we need to > confirm that the modification on master does not make it slower. I'm not sure > but I used a big lock to protect everything in the new implementation, so it > may have bad impact on performance. > Second, some configurations of the old AsyncRpcClient is gone. Such as > "hbase.rpc.client.threads.max" and "hbase.rpc.client.nativetransport". Now > You could pass a EventLoopGroup object directly through a helper class which > makes it more flexible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17763) IPCUtil.wrapException will wrap DoNotRetryIOException with IOException
[
https://issues.apache.org/jira/browse/HBASE-17763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904214#comment-15904214
]
Duo Zhang commented on HBASE-17763:
---
{quote}
Is there a reason to wrap the DNRE in a new one:
{quote}
The wrapException method here aims to add the address to the exception message
for better debugging.
> IPCUtil.wrapException will wrap DoNotRetryIOException with IOException
> --
>
> Key: HBASE-17763
> URL: https://issues.apache.org/jira/browse/HBASE-17763
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17763.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17745) Support short circuit connection for master services
[ https://issues.apache.org/jira/browse/HBASE-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17745: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Support short circuit connection for master services > > > Key: HBASE-17745 > URL: https://issues.apache.org/jira/browse/HBASE-17745 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0 > > Attachments: HBASE-17745.patch, HBASE-17745.v2.patch, > HBASE-17745.v2.trival.patch, HBASE-17745.v2.trival.patch, HBASE-17745.v3.patch > > > As titled, now we have short circuit connection, but no short circuit for > master services, and we propose to support it in this JIRA. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17745) Support short circuit connection for master services
[ https://issues.apache.org/jira/browse/HBASE-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-17745: -- Affects Version/s: 2.0 Hadoop Flags: Reviewed Fix Version/s: 2.0 Pushed into master branch, thanks for review [~stack] [~tedyu] > Support short circuit connection for master services > > > Key: HBASE-17745 > URL: https://issues.apache.org/jira/browse/HBASE-17745 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0 >Reporter: Yu Li >Assignee: Yu Li > Fix For: 2.0 > > Attachments: HBASE-17745.patch, HBASE-17745.v2.patch, > HBASE-17745.v2.trival.patch, HBASE-17745.v2.trival.patch, HBASE-17745.v3.patch > > > As titled, now we have short circuit connection, but no short circuit for > master services, and we propose to support it in this JIRA. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[
https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904196#comment-15904196
]
Ted Yu commented on HBASE-17453:
I got the following compilation errors:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile
(default-testCompile) on project hbase-client: Compilation failure: Compilation
failure:
[ERROR]
/Users/tyu/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:[507,55]
cannot find symbol
[ERROR] symbol: class PingRequest
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos
[ERROR]
/Users/tyu/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:[506,24]
cannot find symbol
[ERROR] symbol: class PingResponse
[ERROR] location: class
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos
[ERROR]
/Users/tyu/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:[505,5]
method does not override or implement a method from a supertype
[ERROR]
/Users/tyu/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:[509,19]
package PingResponse does not exist
[ERROR]
/Users/tyu/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java:[509,43]
cannot find symbol
[ERROR] symbol: variable PingResponse
[ERROR] location: class
org.apache.hadoop.hbase.client.TestClientNoCluster.FakeServer
{code}
Can you double check ?
> add Ping into HBase server for deprecated GetProtocolVersion
>
>
> Key: HBASE-17453
> URL: https://issues.apache.org/jira/browse/HBASE-17453
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 1.2.2
>Reporter: Tianying Chang
>Assignee: Tianying Chang
>Priority: Minor
> Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch
>
>
> Our HBase service is hosted in AWS. We saw cases where the connection between
> the client (Asynchbase in our case) and server stop working but did not throw
> any exception, therefore traffic stuck. So we added a "Ping" feature in
> AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side,
> if no traffic for given time, we send the "Ping", if no response back for
> "Ping", we assume the connect is bad and reconnect.
> Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is
> deprecated. To be able to support same detect/reconnect feature, we added
> Ping() in our internal HBase 1.2 branch, and also patched accordingly in
> Asynchbase 1.7.
> We would like to open source this feature since it is useful for use case in
> AWS environment.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[
https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904188#comment-15904188
]
Hadoop QA commented on HBASE-15597:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 46s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 55s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 28s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857136/HBASE-15597-V3.patch |
| JIRA Issue | HBASE-15597 |
| Optional Tests | asflicense scalac scaladoc unit compile |
| uname | Linux 7e5c444f9276 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1160315 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6047/testReport/ |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6047/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Clean up configuration keys used in hbase-spark module
> --
>
> Key: HBASE-15597
> URL: https://issues.apache.org/jira/browse/HBASE-15597
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Yi Liang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch,
> HBASE-15597-V3.patch
>
>
> This should be considered a blocker for backport to branch-1 since it will
> impact our compatibility.
> The constants we expose in configuration should all start with "hbase". Since
> our configurations keys for the spark integration all relate to that system,
> the prefix for all configuration keys (excluding those cases where we need to
> do something special due to restrictions in how properties are handled by
> e.g. spark) should be "hbase.spark".
> Before publishing a public api labeled version of our spark integration we
> should review all of our configuration keys to make sure they either conform
> to the "hbase.spark" prefix or they have a comment documenting why they need
> to be otherwise.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904181#comment-15904181 ] Ashu Pachauri commented on HBASE-17579: --- Thank you [~ghelmling]! I have added a release note. > Backport HBASE-16302 to 1.3.1 > - > > Key: HBASE-17579 > URL: https://issues.apache.org/jira/browse/HBASE-17579 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.1 > > Attachments: HBASE-17579.branch-1.3.001.patch, > HBASE-17579.branch-1.3.002.patch, HBASE-17579.branch-1.3.003.patch > > > This is a simple enough change to be included in 1.3.1, and replication > monitoring essentially breaks without this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-17579: -- Release Note: The replication sources and replication sinks will now expose ageOfLastShippedOp and ageOfLastAppliedOp metrics respectively as histograms (with percentile suffixes). The 'max' values in these histograms are also exported as simple metrics (no percentile suffix) under the same metric names for backward compatibility. > Backport HBASE-16302 to 1.3.1 > - > > Key: HBASE-17579 > URL: https://issues.apache.org/jira/browse/HBASE-17579 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.1 > > Attachments: HBASE-17579.branch-1.3.001.patch, > HBASE-17579.branch-1.3.002.patch, HBASE-17579.branch-1.3.003.patch > > > This is a simple enough change to be included in 1.3.1, and replication > monitoring essentially breaks without this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17731) Fractional latency reporting in MultiThreadedAction
[ https://issues.apache.org/jira/browse/HBASE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17731: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 1.3.1) Status: Resolved (was: Patch Available) > Fractional latency reporting in MultiThreadedAction > --- > > Key: HBASE-17731 > URL: https://issues.apache.org/jira/browse/HBASE-17731 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17731.patch > > > When average latency is less than one millisecond the LoadTestTool tool > reports a latency of 0. Better to report a fraction out to a couple of > decimal points. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17731) Fractional latency reporting in MultiThreadedAction
[ https://issues.apache.org/jira/browse/HBASE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904180#comment-15904180 ] Andrew Purtell commented on HBASE-17731: Thanks Anoop. Committed > Fractional latency reporting in MultiThreadedAction > --- > > Key: HBASE-17731 > URL: https://issues.apache.org/jira/browse/HBASE-17731 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17731.patch > > > When average latency is less than one millisecond the LoadTestTool tool > reports a latency of 0. Better to report a fraction out to a couple of > decimal points. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17736) Some options can't be configured by the shell
[ https://issues.apache.org/jira/browse/HBASE-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17736: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Some options can't be configured by the shell > - > > Key: HBASE-17736 > URL: https://issues.apache.org/jira/browse/HBASE-17736 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17736.branch-1.v0.patch, HBASE-17736.v0.patch, > HBASE-17736.v1.patch, HBASE-17736.v2.patch > > > The lost options for table are shown below. > # setFlushPolicyClassName > # setPriority > # setRegionMemstoreReplication > # setRegionSplitPolicyClassName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[ https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16977: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to branch-1.3+. > VerifyReplication should log a printable representation of the row keys > --- > > Key: HBASE-16977 > URL: https://issues.apache.org/jira/browse/HBASE-16977 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16977.branch-1.3.001.patch, > HBASE-16977.master.001.patch, HBASE-16977.V1.patch > > > VerifyReplication prints out the row keys for offending rows in the task logs > for the MR job. However, the log is useless if the row key contains non > printable characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904162#comment-15904162 ] Tianying Chang commented on HBASE-17453: [~ted_yu] [~Apache9] [~saint@gmail.com] attached a patch generated from master branch. > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-17453: --- Attachment: HBASE-17453-master.patch > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Attachments: HBASE-17453-1.2.patch, HBASE-17453-master.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904130#comment-15904130 ] Ted Yu commented on HBASE-17460: Partially disabled replication means that replication is disabled on subset of the column families of the table (enabled on other column families). > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, > 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, > HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, > HBASE-17460-addendum2-v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, > HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904122#comment-15904122 ] NITIN VERMA commented on HBASE-17460: - [~tedyu] Please help me understand. What is partially disabled replication mean here? Does it mean the replication disabled on certain CF's? > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, > 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, > HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, > HBASE-17460-addendum2-v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, > HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-15597: - Attachment: HBASE-15597-V3.patch > Clean up configuration keys used in hbase-spark module > -- > > Key: HBASE-15597 > URL: https://issues.apache.org/jira/browse/HBASE-15597 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, > HBASE-15597-V3.patch > > > This should be considered a blocker for backport to branch-1 since it will > impact our compatibility. > The constants we expose in configuration should all start with "hbase". Since > our configurations keys for the spark integration all relate to that system, > the prefix for all configuration keys (excluding those cases where we need to > do something special due to restrictions in how properties are handled by > e.g. spark) should be "hbase.spark". > Before publishing a public api labeled version of our spark integration we > should review all of our configuration keys to make sure they either conform > to the "hbase.spark" prefix or they have a comment documenting why they need > to be otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17737) Thrift2 proxy should support scan timeRange per column family
[ https://issues.apache.org/jira/browse/HBASE-17737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904106#comment-15904106 ] Yechao Chen commented on HBASE-17737: - [~tedyu] Thanks for reviews. > Thrift2 proxy should support scan timeRange per column family > - > > Key: HBASE-17737 > URL: https://issues.apache.org/jira/browse/HBASE-17737 > Project: HBase > Issue Type: New Feature > Components: Thrift >Affects Versions: 2.0.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17737.branch-1.001.patch, > HBASE-17737.branch-1.002.patch, HBASE-17737.master.001.patch, > HBASE-17737.patch, HBASE-17737-v1.patch, HBASE-17737-v2.patch > > > see discussion in HBASE-14872 > and HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15597) Clean up configuration keys used in hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-15597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904107#comment-15904107 ] Yi Liang commented on HBASE-15597: -- change some configuration keys based on TableInputFormat > Clean up configuration keys used in hbase-spark module > -- > > Key: HBASE-15597 > URL: https://issues.apache.org/jira/browse/HBASE-15597 > Project: HBase > Issue Type: Bug > Components: spark >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-15597-v1.patch, HBASE-15597-V2.patch, > HBASE-15597-V3.patch > > > This should be considered a blocker for backport to branch-1 since it will > impact our compatibility. > The constants we expose in configuration should all start with "hbase". Since > our configurations keys for the spark integration all relate to that system, > the prefix for all configuration keys (excluding those cases where we need to > do something special due to restrictions in how properties are handled by > e.g. spark) should be "hbase.spark". > Before publishing a public api labeled version of our spark integration we > should review all of our configuration keys to make sure they either conform > to the "hbase.spark" prefix or they have a comment documenting why they need > to be otherwise. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17460) enable_table_replication can not perform cyclic replication of a table
[ https://issues.apache.org/jira/browse/HBASE-17460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904097#comment-15904097 ] Ted Yu commented on HBASE-17460: Currently HBaseAdmin#isTableRepEnabled() returns false if ANY of the columns families is not replicated, meaning that if you have a table with even one column family without replication you could not disable replication. Admin#disableTableReplication() returns nothing and isTableRepEnabled() returns boolean. For master branch, we can let isTableRepEnabled() return enum (partially disabled, disabled, etc) This way, Admin#disableTableReplication() can return meaningful value to the user. [~Bhupendra] [~ashish singhi] [~nitin.ve...@gmail.com]: What do you think ? > enable_table_replication can not perform cyclic replication of a table > -- > > Key: HBASE-17460 > URL: https://issues.apache.org/jira/browse/HBASE-17460 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: NITIN VERMA >Assignee: NITIN VERMA > Labels: replication > Fix For: 2.0.0, 1.4.0 > > Attachments: 17460-addendum.txt, 17460-addendum.v2.txt, > 17460.branch-1.v3.txt, 17460.v5.txt, HBASE-17460-addendum2.patch, > HBASE-17460-addendum2-v2.patch, HBASE-17460-addendum2-v3.patch, > HBASE-17460-addendum2-v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4.patch, > HBASE-17460-branch-1-v5-addendumv2-addendum2v4-v2.patch, > HBASE-17460-branch-1-v5-plusaddendum-v2.patch, HBASE-17460.patch, > HBASE-17460_v2.patch, HBASE-17460_v3.patch, HBASE-17460_v4.patch > > Original Estimate: 96h > Remaining Estimate: 96h > > The enable_table_replication operation is broken for cyclic replication of > HBase table as we compare all the properties of column families (including > REPLICATION_SCOPE). > Below is exactly what happens: > 1. Running "enable_table_replication 'table1' " opeartion on first cluster > will set the REPLICATION_SCOPE of all column families to peer id '1'. This > will also create a table on second cluster where REPLICATION_SCOPE is still > set to peer id '0'. > 2. Now when we run "enable_table_replication 'table1'" on second cluster, we > compare all the properties of table (including REPLICATION_SCOPE_, which > obviously is different now. > I am proposing a fix for this issue where we should avoid comparing > REPLICATION_SCOPE inside HColumnDescriotor::compareTo() method, especially > when replication is not already enabled on the desired table. > I have made that change and it is working. I will submit the patch soon. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-17453: --- Attachment: HBASE-17453-master.patch > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Attachments: HBASE-17453-1.2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17453) add Ping into HBase server for deprecated GetProtocolVersion
[ https://issues.apache.org/jira/browse/HBASE-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tianying Chang updated HBASE-17453: --- Attachment: (was: HBASE-17453-master.patch) > add Ping into HBase server for deprecated GetProtocolVersion > > > Key: HBASE-17453 > URL: https://issues.apache.org/jira/browse/HBASE-17453 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 1.2.2 >Reporter: Tianying Chang >Assignee: Tianying Chang >Priority: Minor > Attachments: HBASE-17453-1.2.patch > > > Our HBase service is hosted in AWS. We saw cases where the connection between > the client (Asynchbase in our case) and server stop working but did not throw > any exception, therefore traffic stuck. So we added a "Ping" feature in > AsyncHBase 1.5 by utilizing the GetProtocolVersion() API provided at RS side, > if no traffic for given time, we send the "Ping", if no response back for > "Ping", we assume the connect is bad and reconnect. > Now we are upgrading cluster from 94 to 1.2. However, GetProtocolVersion() is > deprecated. To be able to support same detect/reconnect feature, we added > Ping() in our internal HBase 1.2 branch, and also patched accordingly in > Asynchbase 1.7. > We would like to open source this feature since it is useful for use case in > AWS environment. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17763) IPCUtil.wrapException will wrap DoNotRetryIOException with IOException
[ https://issues.apache.org/jira/browse/HBASE-17763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904068#comment-15904068 ] stack commented on HBASE-17763: --- Is there a reason to wrap the DNRE in a new one: 176 return (IOException) new DoNotRetryIOException( 177 "Call to " + addr + " failed on local exception: " + exception).initCause(exception); This is the debug? 1497e.printStackTrace(); +1 on commit after addressing above. > IPCUtil.wrapException will wrap DoNotRetryIOException with IOException > -- > > Key: HBASE-17763 > URL: https://issues.apache.org/jira/browse/HBASE-17763 > Project: HBase > Issue Type: Bug > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-17763.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-17579: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to branch-1.3. [~ashu210890], can you add a release note here? > Backport HBASE-16302 to 1.3.1 > - > > Key: HBASE-17579 > URL: https://issues.apache.org/jira/browse/HBASE-17579 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.1 > > Attachments: HBASE-17579.branch-1.3.001.patch, > HBASE-17579.branch-1.3.002.patch, HBASE-17579.branch-1.3.003.patch > > > This is a simple enough change to be included in 1.3.1, and replication > monitoring essentially breaks without this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17736) Some options can't be configured by the shell
[ https://issues.apache.org/jira/browse/HBASE-17736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904030#comment-15904030 ] Andrew Purtell commented on HBASE-17736: Thanks [~chia7712], lgtm, committing > Some options can't be configured by the shell > - > > Key: HBASE-17736 > URL: https://issues.apache.org/jira/browse/HBASE-17736 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17736.branch-1.v0.patch, HBASE-17736.v0.patch, > HBASE-17736.v1.patch, HBASE-17736.v2.patch > > > The lost options for table are shown below. > # setFlushPolicyClassName > # setPriority > # setRegionMemstoreReplication > # setRegionSplitPolicyClassName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904021#comment-15904021 ] Jerry He commented on HBASE-16598: -- Hi, [~syuanjiang] It is probably for the incompatible change mentioned in the Release Notes. > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598.patch, HBASE-16598-v2.patch, > HBASE-16598-v3.patch, HBASE-16598-v4.patch, HBASE-16598-v5.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904015#comment-15904015 ] Tianying Chang commented on HBASE-17138: [~haoran] [~carp84] This improvement looks super useful to us too. We are currently on HBase 1.2.1. Will be very helpful if this can be backport. Anyone currently working on it? > Backport read-path offheap (HBASE-11425) to branch-1 > > > Key: HBASE-17138 > URL: https://issues.apache.org/jira/browse/HBASE-17138 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Sun > Attachments: > 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, > 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, > 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch > > > From the > [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] > of sharing our experience and performance data of read-path offheap usage in > Alibaba search, we could see people are positive to have HBASE-11425 in > branch-1, so I'd like to create a JIRA and move the discussion and decision > making here. > Echoing some comments from the mail thread: > Bryan: > Is the backported patch available anywhere? If it ends up not getting > officially backported to branch-1 due to 2.0 around the corner, some of us > who build our own deploy may want to integrate into our builds > Andrew: > Yes, please, the patches will be useful to the community even if we decide > not to backport into an official 1.x release. > Enis: > I don't see any reason why we cannot backport to branch-1. > Ted: > Opening a JIRA would be fine. This makes it easier for people to obtain the > patch(es) > Nick: > From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to > support 1.x releases for some time... I would guess these will be > maintained until 2.2 at least. Therefore, offheap patches that have seen > production exposure seem like a reasonable candidate for backport, perhaps in > a 1.4 or 1.5 release timeframe. > Anoop: > Because of some compatibility issues, we decide that this will be done in 2.0 > only.. Ya as Andy said, it would be great to share the 1.x backported > patches. > The following is all the jira ids we have back ported: > HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells > (Ram) > HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and > AbstractHFileReader; ditto for Scanners and BlockReader, etc. > HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner. > HBASE-13450 - Purge RawBytescomparator from the writers and readers for > HBASE-10800 (Ram) > HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo. > HBASE-12048 Remove deprecated APIs from Filter. > HBASE-10800 - Use CellComparator instead of KVComparator (Ram) > HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], > int, int. > HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with > byte[],int,int args in favor of taking Cell arg. > HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int > length) in favor of filterRowKey(Cell firstRowCell). > HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner. > HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], > int, int. > HBASE-11911 Break up tests into more fine grained categories (Alex Newman) > HBASE-12059 Create hbase-annotations module > HBASE-12106 Move test annotations to test artifact (Enis Soztutar) > HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers. > HBASE-15679 Assertion on wrong variable in > TestReplicationThrottler#testThrottling > HBASE-13931 Move Unsafe based operations to UnsafeAccess. > HBASE-12345 Unsafe based ByteBuffer Comparator. > HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int > llength, byte[] right, int roffset, int rlength). > HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn > HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport > this patch) > HBASE-13448 New Cell implementation with cached component offsets/lengths. > HBASE-13387 Add ByteBufferedCell an extension to Cell. > HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum. > HBASE-12650 Move ServerName to hbase-common module (partially backport this > patch) > HBASE-12296 Filters should work with ByteBufferedCell. > HBASE-14120 ByteBufferUtils#compareTo small optimization. > HBASE-13510 - Purge ByteBloomFilter (Ram) > HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be > easy to use in the CellComparators (Ram) >
[jira] [Commented] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[
https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904014#comment-15904014
]
Hadoop QA commented on HBASE-16977:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 98m 46s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 51s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857084/HBASE-16977.master.001.patch
|
| JIRA Issue | HBASE-16977 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7ad9ac8dc4c1 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1160315 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6045/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6045/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> VerifyReplication should log a printable representation of the row keys
> ---
>
> Key: HBASE-16977
> URL: https://issues.apache.org/jira/browse/HBASE-16977
> Project: HBase
> Issue Type: Improvement
> Components:
[jira] [Commented] (HBASE-17764) Solve TestMultiSlaveReplication flakiness
[ https://issues.apache.org/jira/browse/HBASE-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904012#comment-15904012 ] Ted Yu commented on HBASE-17764: +1 > Solve TestMultiSlaveReplication flakiness > -- > > Key: HBASE-17764 > URL: https://issues.apache.org/jira/browse/HBASE-17764 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-17764.v1-branch-1.patch > > > When I doing testing for a feature I am working on, the > TestMultiSlaveReplication test was flaky. > TestMultiSlaveReplication#testMultiSlaveReplication would always pass when > running alone; however, it would fail occasionally when running with the > suite (the other test TestMultiSlaveReplication#testZKLockCleaner always run > first and I suspect that it withhold some zookeeper resource, which caused > the problem). > After I set start and shutdown zk cluster before and after each test, I no > longer see the problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17761) Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of race condition
[ https://issues.apache.org/jira/browse/HBASE-17761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-17761: -- Resolution: Fixed Fix Version/s: 1.4.0 Status: Resolved (was: Patch Available) committed to branch-1 too. Thanks [~uagashe] > Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of > race condition > -- > > Key: HBASE-17761 > URL: https://issues.apache.org/jira/browse/HBASE-17761 > Project: HBase > Issue Type: Bug > Components: Region Assignment >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17761_branch-1.v1.patch, > HBASE-17761-branch-1.v1.patch, HBASE-17761.patch, HBASE-17761.v1.patch > > > After moving the region code waits till all regions are assigned but not for > region to go online on a destination server. On branch-1 TestAdmin1.java has > a function moveRegionAndWait() that moves the region and waits for the region > to become online on destination server. We need to use that function in this > test. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[
https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903949#comment-15903949
]
Hadoop QA commented on HBASE-17579:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 32s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 40s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:66fbe99 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856161/HBASE-17579.branch-1.3.003.patch
|
| JIRA Issue | HBASE-17579 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7e26942ffbf4 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[
https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903927#comment-15903927
]
Hadoop QA commented on HBASE-17765:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 104m 47s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 148m 36s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857073/HBASE-17765-V01.patch
|
| JIRA Issue | HBASE-17765 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 151874f9ead5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 1160315 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6043/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6043/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Reviving the merge possibility in the CompactingMemStore
>
>
> Key: HBASE-17765
> URL: https://issues.apache.org/jira/browse/HBASE-17765
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee:
[jira] [Commented] (HBASE-17764) Solve TestMultiSlaveReplication flakiness
[ https://issues.apache.org/jira/browse/HBASE-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903872#comment-15903872 ] Stephen Yuan Jiang commented on HBASE-17764: Another note: in HBASE-16598 (at this time, only commit in master branch), the TestMultiSlaveReplication#testZKLockCleaner was removed; therefore, not problem in master branch. > Solve TestMultiSlaveReplication flakiness > -- > > Key: HBASE-17764 > URL: https://issues.apache.org/jira/browse/HBASE-17764 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-17764.v1-branch-1.patch > > > When I doing testing for a feature I am working on, the > TestMultiSlaveReplication test was flaky. > TestMultiSlaveReplication#testMultiSlaveReplication would always pass when > running alone; however, it would fail occasionally when running with the > suite (the other test TestMultiSlaveReplication#testZKLockCleaner always run > first and I suspect that it withhold some zookeeper resource, which caused > the problem). > After I set start and shutdown zk cluster before and after each test, I no > longer see the problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903904#comment-15903904 ] Andrew Purtell commented on HBASE-17762: Looking forward to an updated patch that has server side logging with the information I requested, and client side logging too. :-) However that gets done is fine by me > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17764) Solve TestMultiSlaveReplication flakiness
[
https://issues.apache.org/jira/browse/HBASE-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903871#comment-15903871
]
Stephen Yuan Jiang commented on HBASE-17764:
The V1 change is local to {{TestMultiSlaveReplication}} test suite; the
{{org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionSplit}}
test failure has nothing to do with this change.
> Solve TestMultiSlaveReplication flakiness
> --
>
> Key: HBASE-17764
> URL: https://issues.apache.org/jira/browse/HBASE-17764
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Minor
> Attachments: HBASE-17764.v1-branch-1.patch
>
>
> When I doing testing for a feature I am working on, the
> TestMultiSlaveReplication test was flaky.
> TestMultiSlaveReplication#testMultiSlaveReplication would always pass when
> running alone; however, it would fail occasionally when running with the
> suite (the other test TestMultiSlaveReplication#testZKLockCleaner always run
> first and I suspect that it withhold some zookeeper resource, which caused
> the problem).
> After I set start and shutdown zk cluster before and after each test, I no
> longer see the problem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code
[ https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903870#comment-15903870 ] Stephen Yuan Jiang commented on HBASE-16598: [~jerryhe], from description, this also apply to branch-1. Any reason why this improvement not in branch-1? > Enable zookeeper useMulti always and clean up in HBase code > --- > > Key: HBASE-16598 > URL: https://issues.apache.org/jira/browse/HBASE-16598 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-16598.patch, HBASE-16598-v2.patch, > HBASE-16598-v3.patch, HBASE-16598-v4.patch, HBASE-16598-v5.patch > > > We have zookeeper useMulti default true since HBase 1.0.0 > And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0" > The benefit of useMulti is obvious. > Let's clean up the code to remove useMulti false case so that we don't have > to continue with the hassle of maintaining two version of the code (e.g. in > replication) . No go back to pre 3.4.x Zookeeper. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17764) Solve TestMultiSlaveReplication flakiness
[
https://issues.apache.org/jira/browse/HBASE-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903857#comment-15903857
]
Hadoop QA commented on HBASE-17764:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 56s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 47s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 56s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 59s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestSerialReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857069/HBASE-17764.v1-branch-1.patch
|
| JIRA Issue | HBASE-17764 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 112f1873baa1 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 /
[jira] [Commented] (HBASE-17761) Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of race condition
[
https://issues.apache.org/jira/browse/HBASE-17761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903828#comment-15903828
]
Hadoop QA commented on HBASE-17761:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} branch-1 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 50s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 27s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 17s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857066/HBASE-17761-branch-1.v1.patch
|
| JIRA Issue | HBASE-17761 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 6ceecc7daeaa 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1 / c70769e |
| Default Java | 1.7.0_80 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903831#comment-15903831 ] Edward Bortnikov commented on HBASE-16417: -- bq. On the 90th percentile degradation when BASIC, how many segments we talking 2 or 3 or more than this? Taking the liberty of answering for [~eshcar]. The current default active segment size cap for in-memory flush is 1/4 the memstore size cap for disk flush. Which means that the expected number of segments in the pipeline is 4/2=2. However, since disk flush is non-immediate, new segments can sometime pile up, especially under a very high write rate as exercised in our test. We don't have easily trackable metrics installed (maybe should have) but probably we're speaking about many more segments here. The number can't exceed 30 - at that point, a forceful merge happens. We guess that looking up the key in every single segment (to initialize the scan) is what leads to the high tail latency. We're taking a closer look at merge (index compaction only, no data copy), hopefully we'll show there's no material damage about it .. even EAGER does not look too bad .. A matter of a few more days of experimentation. Thanks. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[ https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-16977: -- Attachment: HBASE-16977.branch-1.3.001.patch Patch for master also applies to branch-1 but not to branch-1.3; uploading a patch for 1.3. > VerifyReplication should log a printable representation of the row keys > --- > > Key: HBASE-16977 > URL: https://issues.apache.org/jira/browse/HBASE-16977 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16977.branch-1.3.001.patch, > HBASE-16977.master.001.patch, HBASE-16977.V1.patch > > > VerifyReplication prints out the row keys for offending rows in the task logs > for the MR job. However, the log is useless if the row key contains non > printable characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903801#comment-15903801 ] churro morales commented on HBASE-17762: server side logging sounds like a good idea, [~esteban] if you want to add that optional string to version info that would be great, [~apurtell] what do you think? > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-16977) VerifyReplication should log a printable representation of the row keys
[ https://issues.apache.org/jira/browse/HBASE-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashu Pachauri updated HBASE-16977: -- Attachment: HBASE-16977.master.001.patch Attaching the rebased patch. > VerifyReplication should log a printable representation of the row keys > --- > > Key: HBASE-16977 > URL: https://issues.apache.org/jira/browse/HBASE-16977 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri >Priority: Minor > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: HBASE-16977.master.001.patch, HBASE-16977.V1.patch > > > VerifyReplication prints out the row keys for offending rows in the task logs > for the MR job. However, the log is useless if the row key contains non > printable characters. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17579) Backport HBASE-16302 to 1.3.1
[ https://issues.apache.org/jira/browse/HBASE-17579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903781#comment-15903781 ] Gary Helmling commented on HBASE-17579: --- I manually triggered a pre-commit build to get some test results. > Backport HBASE-16302 to 1.3.1 > - > > Key: HBASE-17579 > URL: https://issues.apache.org/jira/browse/HBASE-17579 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 1.3.1 > > Attachments: HBASE-17579.branch-1.3.001.patch, > HBASE-17579.branch-1.3.002.patch, HBASE-17579.branch-1.3.003.patch > > > This is a simple enough change to be included in 1.3.1, and replication > monitoring essentially breaks without this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16302) age of last shipped op and age of last applied op should be histograms
[
https://issues.apache.org/jira/browse/HBASE-16302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903779#comment-15903779
]
Hadoop QA commented on HBASE-16302:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-16302 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839904/HBASE-16302.patch.v0.patch
|
| JIRA Issue | HBASE-16302 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6044/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> age of last shipped op and age of last applied op should be histograms
> --
>
> Key: HBASE-16302
> URL: https://issues.apache.org/jira/browse/HBASE-16302
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16302.patch.v0.patch
>
>
> Replication exports metric ageOfLastShippedOp as an indication of how much
> replication is lagging. But, with multiwal enabled, it's not representative
> because replication could be lagging for a long time for one wal group
> (something wrong with a particular region) while being fine for others. The
> ageOfLastShippedOp becomes a useless metric for alerting in such a case.
> Also, since there is no mapping between individual replication sources and
> replication sinks, the age of last applied op can be a highly spiky metric if
> only certain replication sources are lagging.
> We should use histograms for these metrics and use maximum value of this
> histogram to report replication lag when building stats.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903767#comment-15903767 ] Esteban Gutierrez edited comment on HBASE-17762 at 3/9/17 8:13 PM: --- Also, looks like the change is only affecting client side logging and I think we should have this in the server side too. One thing I've been considering too is that we should add an optional string to VersionInfo to provide additional information to identify the client (think of something similar to use mapreduce.task.attempt.id to identify the clients in HDFS). If that sounds interesting for you guys I'd be happy to work in a patch for that. was (Author: esteban): Also, looks like the change is only affecting client side logging and I think we should have this in the server side too. One thing I've been considering too is that we should add an optional string to VersionInfo two provide additional information to identify the client (think of something similar to use mapreduce.task.attempt.id to identify the clients in HDFS). If that sounds interesting for you guys I'd be happy to work in a patch for that. > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903772#comment-15903772 ] Andrew Purtell commented on HBASE-17762: +1 to server side logging. My comment implied it sorry that wasn't clear. And keep the client side logging too > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903769#comment-15903769 ] stack commented on HBASE-14070: --- [~churromorales] Man. Would be cool if we could get this in. Our [~ranuva] is busy finishing up at school. I'm sure he is interested but probably doesn't have time. Would a meeting on state help [~churromorales]? We could rope Sai in (and probably our HLC granddad, [~enis]). > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903767#comment-15903767 ] Esteban Gutierrez commented on HBASE-17762: --- Also, looks like the change is only affecting client side logging and I think we should have this in the server side too. One thing I've been considering too is that we should add an optional string to VersionInfo two provide additional information to identify the client (think of something similar to use mapreduce.task.attempt.id to identify the clients in HDFS). If that sounds interesting for you guys I'd be happy to work in a patch for that. > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15716) HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random read
[ https://issues.apache.org/jira/browse/HBASE-15716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903764#comment-15903764 ] stack commented on HBASE-15716: --- Thanks for reviving this one [~vincentpoon] > HRegion#RegionScannerImpl scannerReadPoints synchronization constrains random > read > -- > > Key: HBASE-15716 > URL: https://issues.apache.org/jira/browse/HBASE-15716 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack > Attachments: > 15716.implementation.using.ScannerReadPoints.branch-1.patch, > 15716.prune.synchronizations.patch, 15716.prune.synchronizations.v3.patch, > 15716.prune.synchronizations.v4.patch, 15716.prune.synchronizations.v4.patch, > 15716.wip.more_to_be_done.patch, before_after.png, > current-branch-1.vs.NoSynchronization.vs.Patch.png, > HBASE-15716.branch-1.001.patch, HBASE-15716.branch-1.002.patch, > HBASE-15716.branch-1.003.patch, HBASE-15716.branch-1.004.patch, > HBASE-15716.branch-1.005.patch, hits.png, remove_cslm.patch, > remove.locks.patch, ScannerReadPoints.java, ScannerReadPoints.v2.java, Screen > Shot 2016-04-26 at 2.05.45 PM.png, Screen Shot 2016-04-26 at 2.06.14 PM.png, > Screen Shot 2016-04-26 at 2.07.06 PM.png, Screen Shot 2016-04-26 at 2.25.26 > PM.png, Screen Shot 2016-04-26 at 6.02.29 PM.png, Screen Shot 2016-04-27 at > 9.49.35 AM.png, Screen Shot 2016-06-30 at 9.52.52 PM.png, Screen Shot > 2016-06-30 at 9.54.08 PM.png, TestScannerReadPointDoesntGoBack.java, > TestScannerReadPoints.java > > > Here is a [~lhofhansl] special. > When we construct the region scanner, we get our read point and then store it > with the scanner instance in a Region scoped CSLM. This is done under a > synchronize on the CSLM. > This synchronize on a region-scoped Map creating region scanners is the > outstanding point of lock contention according to flight recorder (My work > load is workload c, random reads). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903757#comment-15903757 ] churro morales commented on HBASE-17762: didn't add any new tests because i simply adding some logging statements in HBaseAdmin. > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16417) In-Memory MemStore Policy for Flattening and Compactions
[ https://issues.apache.org/jira/browse/HBASE-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903755#comment-15903755 ] stack commented on HBASE-16417: --- bq. Any chance of using larger values (e.g. 2KB) ? Please be explicit on what you are asking for [~ted_yu] Asking for 'larger' values w/o why or fit criteria you would see satisfied is like asking for values that are 'ethical' or values that smell sweet. [~eshcar] Nice writeup. Those numbers are looking good (I like the write amplification improvements and the less GC...). (Trying to channel [~ted_yu]), how you think the numbers would change if 'default' 1k values were used? Would we see the same benefit do you think? On the 90th percentile degradation when BASIC, how many segments we talking 2 or 3 or more than this? In another issue we might dig in on why the degradation... I'd think read from Segments in parallel should be nice and prompt. Probably something dumb we are doing. For another issue. You thinking the numbers good enough to enable BASIC as default? Merging means more memory churn, more CPU expended would be cool if we could do w/o merge... Nice writeup. > In-Memory MemStore Policy for Flattening and Compactions > > > Key: HBASE-16417 > URL: https://issues.apache.org/jira/browse/HBASE-16417 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-16417-benchmarkresults-20161101.pdf, > HBASE-16417-benchmarkresults-20161110.pdf, > HBASE-16417-benchmarkresults-20161123.pdf, > HBASE-16417-benchmarkresults-20161205.pdf, > HBASE-16417-benchmarkresults-20170309.pdf > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903745#comment-15903745 ] Andrew Purtell edited comment on HBASE-17762 at 3/9/17 7:55 PM: I don't think the current patch is sufficient [~churromorales]. We want (at least): - What action was initiated - When was the action initiated - On what was the action initiated (e.g. region) - Where was the action initiated: suppose the logline will contain this information, everyone will have log4j formatting configured to include the server host in the logline - Who initiated the action: effective user, client IP address Bonus - When the action completed and how long it took in this same logline (for convenience) was (Author: apurtell): I don't think the current patch is sufficient [~churromorales]. We want (at least): - What action was initiated - When was the action initiated - Where was the action initiated: suppose the logline will contain this information, everyone will have log4j formatting configured to include the server host in the logline - Who initiated the action: effective user, client IP address Bonus - When the action completed and how long it took in this same logline (for convenience) > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-14070) Hybrid Logical Clocks for HBase
[ https://issues.apache.org/jira/browse/HBASE-14070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903747#comment-15903747 ] churro morales commented on HBASE-14070: this is so close to being done. So many hbase issues can be resolved with this feature. I would like to try and finish this but I don't want to step on anybody's toes since [~saitejar] is so far along with this patch :). Are you planning on completing the last few open issues? If there is nobody working on this, I would love to try and get this in to HBase 2.x because it could solve so many of our issues. > Hybrid Logical Clocks for HBase > --- > > Key: HBASE-14070 > URL: https://issues.apache.org/jira/browse/HBASE-14070 > Project: HBase > Issue Type: New Feature >Reporter: Enis Soztutar >Assignee: Sai Teja Ranuva > Attachments: HBASE-14070.master.001.patch, > HybridLogicalClocksforHBaseandPhoenix.docx, > HybridLogicalClocksforHBaseandPhoenix.pdf > > > HBase and Phoenix uses systems physical clock (PT) to give timestamps to > events (read and writes). This works mostly when the system clock is strictly > monotonically increasing and there is no cross-dependency between servers > clocks. However we know that leap seconds, general clock skew and clock drift > are in fact real. > This jira proposes using Hybrid Logical Clocks (HLC) as an implementation of > hybrid physical clock + a logical clock. HLC is best of both worlds where it > keeps causality relationship similar to logical clocks, but still is > compatible with NTP based physical system clock. HLC can be represented in > 64bits. > A design document is attached and also can be found here: > https://docs.google.com/document/d/1LL2GAodiYi0waBz5ODGL4LDT4e_bXy8P9h6kWC05Bhw/edit# -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903745#comment-15903745 ] Andrew Purtell commented on HBASE-17762: I don't think the current patch is sufficient [~churromorales]. We want (at least): - What action was initiated - When was the action initiated - Where was the action initiated: suppose the logline will contain this information, everyone will have log4j formatting configured to include the server host in the logline - Who initiated the action: effective user, client IP address Bonus - When the action completed and how long it took in this same logline (for convenience) > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[ https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-17762: --- Status: Open (was: Patch Available) > Add logging to HBaseAdmin for user initiated tasks > -- > > Key: HBASE-17762 > URL: https://issues.apache.org/jira/browse/HBASE-17762 > Project: HBase > Issue Type: Task >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.4.0, 0.98.25 > > Attachments: HBASE-17762.patch, HBASE-17762.v1.patch > > > Things like auditing a forced major compaction are really useful and right > now there is no logging when this is triggered. Other actions may require > logging as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-17766) Generate Javadoc for hbase-spark module
Yi Liang created HBASE-17766: Summary: Generate Javadoc for hbase-spark module Key: HBASE-17766 URL: https://issues.apache.org/jira/browse/HBASE-17766 Project: HBase Issue Type: Bug Reporter: Yi Liang Assignee: Yi Liang Fix For: 2.0.0 Scala classes in hbase-spark module aren't showing up in our API docs nor our internal API docs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Issue Comment Deleted] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-17766: - Comment: was deleted (was: if this idea is ok, i will continue my patch. ) > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17762) Add logging to HBaseAdmin for user initiated tasks
[
https://issues.apache.org/jira/browse/HBASE-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903682#comment-15903682
]
Hadoop QA commented on HBASE-17762:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 34s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
43m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 33s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12857061/HBASE-17762.v1.patch |
| JIRA Issue | HBASE-17762 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0128cd1f6055 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 1160315 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6039/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6039/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add logging to HBaseAdmin for user initiated tasks
> --
>
> Key: HBASE-17762
> URL: https://issues.apache.org/jira/browse/HBASE-17762
> Project: HBase
> Issue Type: Task
>Reporter: churro morales
>Assignee: churro morales
> Fix
[jira] [Commented] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903692#comment-15903692 ] Anastasia Braginsky commented on HBASE-17765: - [~anoop.hbase], [~ram_krish], [~eshcar], [~ebortnik], please take a look on the review board: https://reviews.apache.org/r/57476/ Would like to hear what do you think! > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903684#comment-15903684 ] Yi Liang commented on HBASE-17766: -- The reason why the scala source code API does not show up in javadoc is that our maven-javadoc-plugin does not recognize the .scala file, so it will only generate javadoc for the .java file under hbase-spark. After investigate how spark generate javadoc for their scala mixed project, I found they use com.typesafe.genjavadoc plugin to generate javadoc for their scala file. The approach the plugin taken here is to use the Scala compiler’s support as far as possible and then generate mostly valid Java code corresponding to the AST—but only the class and method structure without implementations, it means for each scala file, there is a corresponding java file generated but without implementation. And then use the maven-javadoc-plugin to generate javadoc based on each generated java code file. and there is 1 limitation for this plugin: In our current hbase javadoc, only @InterfaceAudience.public classes are shown. however this genjavadoc plugin does not recognize customized annotation in our project, when it generate java code, it will ignore those annotation in the scala code. But, this genjavadoc plugin can filter out private class in scala, We can change some public class with @InterfaceAudience.private to private Scala class, fortunately, in scala, even if we define some class as private, this private class can also be used in certain scope. For example private[spark] class A, it means this class A can be used under org.hadoop.hbase.spark package. And this method could filter most private class out of javadoc, but may not apply to only a few classes(1 or 2). And this priavet[scope] is also spark code style. > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17766) Generate Javadoc for hbase-spark module
[ https://issues.apache.org/jira/browse/HBASE-17766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903689#comment-15903689 ] Yi Liang commented on HBASE-17766: -- if this idea is ok, i will continue my patch. > Generate Javadoc for hbase-spark module > > > Key: HBASE-17766 > URL: https://issues.apache.org/jira/browse/HBASE-17766 > Project: HBase > Issue Type: Bug >Reporter: Yi Liang >Assignee: Yi Liang > Fix For: 2.0.0 > > > Scala classes in hbase-spark module aren't showing up in our API docs nor > our internal API docs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17765) Reviving the merge possibility in the CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-17765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Braginsky updated HBASE-17765: Attachment: HBASE-17765-V01.patch > Reviving the merge possibility in the CompactingMemStore > > > Key: HBASE-17765 > URL: https://issues.apache.org/jira/browse/HBASE-17765 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-17765-V01.patch > > > According to the new performance results presented in the HBASE-16417 we see > that the read latency of the 90th percentile of the BASIC policy is too big > due to the need to traverse through too many segments in the pipeline. In > this JIRA we correct the bug in the merge sizing calculations and allow > pipeline size threshold to be a configurable parameter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
