[jira] [Commented] (HBASE-8519) Backup master will never come up if primary master dies during initialization
[
https://issues.apache.org/jira/browse/HBASE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655179#comment-13655179
]
Jerry He commented on HBASE-8519:
-
The missing part seems to be this.
The backup master only check the clusterStateZNode once at the end when it
wakes up and tries to be master again.
It is not notified if the clusterStateZNode has ever been up or not before that.
The real 'stop' case is: clusterStateZNode up --> clusterStateZNode down
The backup master needs to watch for clusterStateZNode. If it is not aware
clusterStateZNode has ever been up
or was up in its whole life, it should treat it as not a real 'stop' case at
the end.
The if condition would become:
{code}
..
if (clusterHasBeenUp && !clusterStatusTracker.isClusterUp()) {
this.master.stop(
"Cluster went down before this master became active");
}
..
{code}
Comments, please.
> Backup master will never come up if primary master dies during initialization
> -
>
> Key: HBASE-8519
> URL: https://issues.apache.org/jira/browse/HBASE-8519
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.7, 0.95.0
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.98.0
>
>
> The problem happens if primary master dies after becoming master but before
> it completes initialization and calls clusterStatusTracker.setClusterUp(),
> The backup master will try to become the master, but will shutdown itself
> promptly because it sees 'the cluster is not up'.
> This is the backup master log:
> 2013-05-09 15:08:05,568 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2013-05-09 15:08:05,573 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster
> started in backup mode. Stalling until master znode is written.
> 2013-05-09 15:08:05,589 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
> already exists and this is not a retry
> 2013-05-09 15:08:05,590 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
> /hbase/backup-masters/xxx.com,6,1368137285373 in backup master directory
> 2013-05-09 15:08:05,595 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the
> active master, xxx.com,6,1368137283107; waiting to become the next active
> master
> 2013-05-09 15:09:45,006 DEBUG
> org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
> Notifying waiting threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
> went down before this master became active
> 2013-05-09 15:09:45,006 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 6
>
> In ActiveMasterManager::blockUntilBecomingActiveMaster()
> {code}
> ..
> if (!clusterStatusTracker.isClusterUp()) {
> this.master.stop(
> "Cluster went down before this master became active");
> }
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8453) TestImportExport failing again due to configuration issues
[
https://issues.apache.org/jira/browse/HBASE-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655178#comment-13655178
]
Andrew Purtell commented on HBASE-8453:
---
I have found that getting a JobConf from the MiniMRCluster and then using
HBaseConfiguration.merge to merge in HBase minicluster details will fix all
remaining Hadoop 2 test failures involving mapreduce. It's just a matter of
finding all of the places Job is instantaned and doing so. Unfortunately the
result then fails for Hadoop 1 but only in EC2 environments. Might get back to
that this week.
> TestImportExport failing again due to configuration issues
> --
>
> Key: HBASE-8453
> URL: https://issues.apache.org/jira/browse/HBASE-8453
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, test
>Affects Versions: 0.98.0, 0.94.8, 0.95.1
>Reporter: Andrew Purtell
> Fix For: 0.98.0, 0.94.8, 0.95.1
>
> Attachments: 8453.patch, 8453-v1-0.94.patch
>
>
> TestImportExport fails for me with a connection refused exception:
> {noformat}
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:162)
> at
> org.apache.hadoop.yarn.client.YarnClientImpl.getNewApplication(YarnClientImpl.java:121)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:107)
> at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:231)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:352)
> [...]
> Caused by: com.google.protobuf.ServiceException: java.net.ConnectException:
> Call From ip-10-174-75-236/10.174.75.236 to 0.0.0.0:8032 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see: http://wiki.apache.org/hadoop/ConnectionRefused
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)
> at com.sun.proxy.$Proxy89.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:159)
> ... 42 more
> {noformat}
> Settings in the MiniMRCluster configuration are not properly propagated in
> this test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8497) Protobuf WAL also needs a trailer
[
https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655176#comment-13655176
]
Hadoop QA commented on HBASE-8497:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582770/HBASE-8497-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5639//console
This message is automatically generated.
> Protobuf WAL also needs a trailer
> --
>
> Key: HBASE-8497
> URL: https://issues.apache.org/jira/browse/HBASE-8497
> Project: HBase
> Issue Type: Sub-task
> Components: Protobufs, wal
>Affects Versions: 0.95.1
>Reporter: Enis Soztutar
>Assignee: Himanshu Vashishtha
> Fix For: 0.98.0, 0.95.1
>
> Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch
>
>
> New Protobuf WAL has a header, but we will probably need a trailer as well,
> reserved for later usage.
> Right now, we can we just serialize an empty trailer, but putting more
> metadata there, like range of sequence_id's, region names, table names etc
> might be needed in the future.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8531) TestZooKeeper fails in trunk/0.95 builds
stack created HBASE-8531:
Summary: TestZooKeeper fails in trunk/0.95 builds
Key: HBASE-8531
URL: https://issues.apache.org/jira/browse/HBASE-8531
Project: HBase
Issue Type: Bug
Components: Zookeeper
Reporter: stack
Assignee: stack
Fix For: 0.95.1
TestZooKeeper fails on occasion. I caught a good example recently. See below
failure stack trace.
It took me a while. I thought the issue had to do w/ our recent ipc
refactorings but it looks like a problem we have always had. In short,
MetaScanner is not handling DoNotRetryIOEs -- it is letting them out. DNRIOEs
when scanning are supposed to force a reset of the scan. HTable#next catches
these and does the necessary scanner reset up. MetaScanner is running some
subset of what HTable does when it is scanning except the part where it catches
a DNRIOE and redoes the scan. Odd.
TestZooKeeper failed in this instance because the test kills a regionserver at
same time as we are trying to create a table. In create table we do a meta
scan using MetaScanner passing a Visitor. The scan starts and gets a
RegionServerStoppedException (This is NOT a DNRIOE -- it should be -- but later
we convert it into one up in ScannerCallable).
DNRIOEs are thrown to the upper layers to handle
Let me look into having MetaScanner just use HTable scanning. It makes an
instance just to find where to start the scan... let me try using this instance
for actually scanning.
TODO: Do this convertion everywhere a DNRIOE could come out.
Here is the stack trace
{code}
org.apache.hadoop.hbase.exceptions.DoNotRetryIOException: Reset scanner
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:209)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:52)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:170)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:212)
at
org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:52)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:131)
at
org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:128)
at
org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:398)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:128)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:103)
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:81)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:448)
at
org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:348)
at
org.apache.hadoop.hbase.TestZooKeeper.testSanity(TestZooKeeper.java:242)
at
org.apache.hadoop.hbase.TestZooKeeper.testRegionServerSessionExpired(TestZooKeeper.java:203)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at
java.util.concurrent.Executor
[jira] [Updated] (HBASE-8497) Protobuf WAL also needs a trailer
[ https://issues.apache.org/jira/browse/HBASE-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8497: --- Attachment: HBASE-8497-v2.patch Incorporated feedback on the initial version. Apart from the nits that are fixed: a) Modified exception handling as suggested. b) Added sanity checks on the trailer size. Currently, this patch makes trailer_max_size to 1MB, and didn't make it configurable; This is to avoid cases where a user lowers the max size and a reader tries to read old logs. Moreover, I see 1MB size a good enough size for the use case (6774). If there is any requirement to increase it, it could be done in some other jira. Please suggest if you think differently. c) Trailer is instantiated in the init() method. While writing the trailer, a check is made in case trailer is set to null or its size exceeds max_size. In this case, a default trailer is written rather with a WARN message. d) Added test case for corrupt trailer. Test: I tested it on my local by killing the RS and doing the split, etc. Ran unit tests for classes HLog* many times. > Protobuf WAL also needs a trailer > -- > > Key: HBASE-8497 > URL: https://issues.apache.org/jira/browse/HBASE-8497 > Project: HBase > Issue Type: Sub-task > Components: Protobufs, wal >Affects Versions: 0.95.1 >Reporter: Enis Soztutar >Assignee: Himanshu Vashishtha > Fix For: 0.98.0, 0.95.1 > > Attachments: HBASE-8497-v0.patch, HBASE-8497-v2.patch > > > New Protobuf WAL has a header, but we will probably need a trailer as well, > reserved for later usage. > Right now, we can we just serialize an empty trailer, but putting more > metadata there, like range of sequence_id's, region names, table names etc > might be needed in the future. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8530) Refine error message from ExportSnapshot when there is leftover snapshot in target cluster
[
https://issues.apache.org/jira/browse/HBASE-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655147#comment-13655147
]
Hadoop QA commented on HBASE-8530:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582763/8530-v1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5638//console
This message is automatically generated.
> Refine error message from ExportSnapshot when there is leftover snapshot in
> target cluster
> --
>
> Key: HBASE-8530
> URL: https://issues.apache.org/jira/browse/HBASE-8530
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 8530-v1.txt
>
>
> ExportSnapshot performs the following validation check before starting export:
> {code}
> // Check if the snapshot already in-progress
> if (outputFs.exists(snapshotTmpDir)) {
> System.err.println("A snapshot with the same name '" + snapshotName +
> "' is in-progress");
> return 1;
> }
> {code}
> The intention was to prevent concurrent snapshot export when snapshot with
> the same name may be outstanding.
> Here is sample output:
> {code}
> hbase@ip-10-165-36-226 root$ hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapone -copy-to
> hdfs://10.12.113.22:8020/apps/hbase/data -mappers 1
> A snapshot with the same name 'snapone' is in-progress
> {code}
> We should note the actual path in the error message so that user can check
> whether the snapshot with same name is still pending.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655148#comment-13655148 ] binlijin commented on HBASE-8450: - HLog - hbase.regionserver.hlog.blocksize and dfs.block.size = 128M or 256M - io.bytes.per.checksum = 4096 > Update hbase-default.xml and general recommendations to better suit current > hw, h2, experience, etc. > > > Key: HBASE-8450 > URL: https://issues.apache.org/jira/browse/HBASE-8450 > Project: HBase > Issue Type: Task > Components: Usability >Reporter: stack >Priority: Critical > Fix For: 0.95.1 > > Attachments: 8450.txt > > > This is a critical task we need to do before we release; review our defaults. > On cursory review, there are configs in hbase-default.xml that no longer have > matching code; there are some that should be changed because we know better > now and others that should be amended because hardware and deploys are bigger > than they used to be. > We could also move stuff around so that the must-edit stuff is near the top > (zk quorum config. is mid-way down the page) and beef up the descriptions -- > especially since these descriptions shine through in the refguide. > Lastly, I notice that our tgz does not "include" an hbase-default.xml other > than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655139#comment-13655139 ] Lars Hofhansl commented on HBASE-5930: -- Added release notes. I also would like add an option to disable this feature (for example by setting base.regionserver.optionalcacheflushinterval to a value <= 0). > Limits the amount of time an edit can live in the memstore. > --- > > Key: HBASE-5930 > URL: https://issues.apache.org/jira/browse/HBASE-5930 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Devaraj Das > Fix For: 0.98.0, 0.94.8, 0.95.1 > > Attachments: 5930-0.94-added-addendum.txt, 5930-0.94.txt, > 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, > 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, > hbase-5930-addendum2.patch, hbase-5930-test-execution.log > > > A colleague of mine ran into an interesting issue. > He inserted some data with the WAL disabled, which happened to fit in the > aggregate Memstores memory. > Two weeks later he a had problem with the HDFS cluster, which caused the > region servers to abort. He found that his data was lost. Looking at the log > we found that the Memstores were not flushed at all during these two weeks. > Should we have an option to flush memstores periodically. There are obvious > downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5930) Limits the amount of time an edit can live in the memstore.
[ https://issues.apache.org/jira/browse/HBASE-5930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5930: - Release Note: This feature limits the time that unflushed data will stay in the memstore. By default a memstore will flush if data older than 1h (360ms) is present. This can be configured via hbase.regionserver.optionalcacheflushinterval (default value is 360). > Limits the amount of time an edit can live in the memstore. > --- > > Key: HBASE-5930 > URL: https://issues.apache.org/jira/browse/HBASE-5930 > Project: HBase > Issue Type: Improvement >Reporter: Lars Hofhansl >Assignee: Devaraj Das > Fix For: 0.98.0, 0.94.8, 0.95.1 > > Attachments: 5930-0.94-added-addendum.txt, 5930-0.94.txt, > 5930-1.patch, 5930-2.1.patch, 5930-2.2.patch, 5930-2.3.patch, 5930-2.4.patch, > 5930-track-oldest-sample.txt, 5930-wip.patch, HBASE-5930-ADD-0.patch, > hbase-5930-addendum2.patch, hbase-5930-test-execution.log > > > A colleague of mine ran into an interesting issue. > He inserted some data with the WAL disabled, which happened to fit in the > aggregate Memstores memory. > Two weeks later he a had problem with the HDFS cluster, which caused the > region servers to abort. He found that his data was lost. Looking at the log > we found that the Memstores were not flushed at all during these two weeks. > Should we have an option to flush memstores periodically. There are obvious > downsides to this, like many small storefiles, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655137#comment-13655137 ] Lars Hofhansl commented on HBASE-6134: -- Patch probably went stale. We never got closure in the 0.94 thing. [~enis], [~sershe], you want this in 0.94? What's your confidence level in this patch? > Improvement for split-worker to speed up distributed log splitting > -- > > Key: HBASE-6134 > URL: https://issues.apache.org/jira/browse/HBASE-6134 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.95.0 > > Attachments: 6134v4.patch, HBASE-6134.patch, HBASE-6134v2.patch, > HBASE-6134v3-92.patch, HBASE-6134v3.patch, HBASE-6134v4-94.patch, > HBASE-6134v4-94v2.patch, HBASE-6134v4.patch > > > First，we do the test between local-master-splitting and > distributed-log-splitting > Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths > splitting work), 400 regions in one hlog file > local-master-split:60s+ > distributed-log-splitting:165s+ > In fact, in our production environment, distributed-log-splitting also took > 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) > We found split-worker split one log file took about 20s > (30ms~50ms per writer.close(); 10ms per create writers ) > I think we could do the improvement for this: > Parallelizing the create and close writers in threads > In the patch, change the logic for distributed-log-splitting same as the > local-master-splitting and parallelizing the close in threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8462) Custom timestamps should not be allowed to be negative
[
https://issues.apache.org/jira/browse/HBASE-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655132#comment-13655132
]
Hadoop QA commented on HBASE-8462:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582762/hbase-8462_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5637//console
This message is automatically generated.
> Custom timestamps should not be allowed to be negative
> --
>
> Key: HBASE-8462
> URL: https://issues.apache.org/jira/browse/HBASE-8462
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.94.8, 0.95.1
>
> Attachments: hbase-8462_v1.patch, hbase-8462_v2.patch
>
>
> Client supplied timestamps should not be allowed to be negative, otherwise
> unpredictable results will follow. Especially, since we are encoding the ts
> using Bytes.Bytes(long), negative timestamps are sorted after positive ones.
> Plus, the new PB messages define ts' as uint64.
> Credit goes to Huned Lokhandwala for reporting this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8462) Custom timestamps should not be allowed to be negative
[
https://issues.apache.org/jira/browse/HBASE-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655130#comment-13655130
]
Ted Yu commented on HBASE-8462:
---
{code}
+if (timestamp < 0) {
+ throw new IllegalArgumentException("Timestamp cannot be negative");
{code}
Giving users more clue would help them pinpoint the problem. Please include
timestamp in the message.
{code}
+if (minStamp < 0 || maxStamp < 0) {
+ throw new IllegalArgumentException("Timestamp cannot be negative");
+}
{code}
Please be specific which timestamp is negative.
Please fill out Release Notes for this change.
Good job.
> Custom timestamps should not be allowed to be negative
> --
>
> Key: HBASE-8462
> URL: https://issues.apache.org/jira/browse/HBASE-8462
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.94.8, 0.95.1
>
> Attachments: hbase-8462_v1.patch, hbase-8462_v2.patch
>
>
> Client supplied timestamps should not be allowed to be negative, otherwise
> unpredictable results will follow. Especially, since we are encoding the ts
> using Bytes.Bytes(long), negative timestamps are sorted after positive ones.
> Plus, the new PB messages define ts' as uint64.
> Credit goes to Huned Lokhandwala for reporting this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8375) Streamline Table durability settings
[
https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655129#comment-13655129
]
Hadoop QA commented on HBASE-8375:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582760/hbase-8375_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestFullLogReconstruction
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:855)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5636//console
This message is automatically generated.
> Streamline Table durability settings
>
>
> Key: HBASE-8375
> URL: https://issues.apache.org/jira/browse/HBASE-8375
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8375_v1.patch, hbase-8375_v2.patch
>
>
> HBASE-7801 introduces the notion of per mutation fine grained durability
> settings.
> This issue is to consider and the discuss the same for the per table settings
> (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose
> the following setting per table:
> * SKIP_WAL (i.e. an unlogged table)
> * ASYNC_WAL (the current deferred log flush)
> * SYNC_WAL (the current default)
> * FSYNC_WAL (for future uses of HDFS' hsync())
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8530) Refine error message from ExportSnapshot when there is leftover snapshot in target cluster
[
https://issues.apache.org/jira/browse/HBASE-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-8530:
--
Fix Version/s: 0.98.0
Status: Patch Available (was: Open)
> Refine error message from ExportSnapshot when there is leftover snapshot in
> target cluster
> --
>
> Key: HBASE-8530
> URL: https://issues.apache.org/jira/browse/HBASE-8530
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 0.98.0
>
> Attachments: 8530-v1.txt
>
>
> ExportSnapshot performs the following validation check before starting export:
> {code}
> // Check if the snapshot already in-progress
> if (outputFs.exists(snapshotTmpDir)) {
> System.err.println("A snapshot with the same name '" + snapshotName +
> "' is in-progress");
> return 1;
> }
> {code}
> The intention was to prevent concurrent snapshot export when snapshot with
> the same name may be outstanding.
> Here is sample output:
> {code}
> hbase@ip-10-165-36-226 root$ hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapone -copy-to
> hdfs://10.12.113.22:8020/apps/hbase/data -mappers 1
> A snapshot with the same name 'snapone' is in-progress
> {code}
> We should note the actual path in the error message so that user can check
> whether the snapshot with same name is still pending.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8530) Refine error message from ExportSnapshot when there is leftover snapshot in target cluster
[
https://issues.apache.org/jira/browse/HBASE-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-8530:
--
Attachment: 8530-v1.txt
> Refine error message from ExportSnapshot when there is leftover snapshot in
> target cluster
> --
>
> Key: HBASE-8530
> URL: https://issues.apache.org/jira/browse/HBASE-8530
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 8530-v1.txt
>
>
> ExportSnapshot performs the following validation check before starting export:
> {code}
> // Check if the snapshot already in-progress
> if (outputFs.exists(snapshotTmpDir)) {
> System.err.println("A snapshot with the same name '" + snapshotName +
> "' is in-progress");
> return 1;
> }
> {code}
> The intention was to prevent concurrent snapshot export when snapshot with
> the same name may be outstanding.
> Here is sample output:
> {code}
> hbase@ip-10-165-36-226 root$ hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapone -copy-to
> hdfs://10.12.113.22:8020/apps/hbase/data -mappers 1
> A snapshot with the same name 'snapone' is in-progress
> {code}
> We should note the actual path in the error message so that user can check
> whether the snapshot with same name is still pending.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8530) Refine error message from ExportSnapshot when there is leftover snapshot in target cluster
Ted Yu created HBASE-8530:
-
Summary: Refine error message from ExportSnapshot when there is
leftover snapshot in target cluster
Key: HBASE-8530
URL: https://issues.apache.org/jira/browse/HBASE-8530
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
ExportSnapshot performs the following validation check before starting export:
{code}
// Check if the snapshot already in-progress
if (outputFs.exists(snapshotTmpDir)) {
System.err.println("A snapshot with the same name '" + snapshotName + "'
is in-progress");
return 1;
}
{code}
The intention was to prevent concurrent snapshot export when snapshot with the
same name may be outstanding.
Here is sample output:
{code}
hbase@ip-10-165-36-226 root$ hbase
org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snapone -copy-to
hdfs://10.12.113.22:8020/apps/hbase/data -mappers 1
A snapshot with the same name 'snapone' is in-progress
{code}
We should note the actual path in the error message so that user can check
whether the snapshot with same name is still pending.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8519) Backup master will never come up if primary master dies during initialization
[
https://issues.apache.org/jira/browse/HBASE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655117#comment-13655117
]
Jerry He commented on HBASE-8519:
-
bq. I meant the reverse for the shutdown znode.
Yes, that all make sense now.
I am able to re-create the problem easily by killing the master right after
issuing the stop-hbase. Then re-start the master before the ephemeral master
znode goes away.
2013-05-10 18:20:17,690 INFO
org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2013-05-10 18:20:17,704 INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
already exists and this is not a retry
2013-05-10 18:20:17,705 INFO
org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
/hbase/backup-masters/xxx.com,6,1368235217501 in backup master directory
2013-05-10 18:20:17,711 INFO
org.apache.hadoop.hbase.master.ActiveMasterManager: Current master has this
master's address, xxx.com,6,1368234912934; master was restarted? Deleting
node.
2013-05-10 18:20:17,711 DEBUG
org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
Notifying waiting threads
2013-05-10 18:20:17,711 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
went down before this master became active
2013-05-10 18:20:17,711 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping
service threads
> Backup master will never come up if primary master dies during initialization
> -
>
> Key: HBASE-8519
> URL: https://issues.apache.org/jira/browse/HBASE-8519
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.7, 0.95.0
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.98.0
>
>
> The problem happens if primary master dies after becoming master but before
> it completes initialization and calls clusterStatusTracker.setClusterUp(),
> The backup master will try to become the master, but will shutdown itself
> promptly because it sees 'the cluster is not up'.
> This is the backup master log:
> 2013-05-09 15:08:05,568 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2013-05-09 15:08:05,573 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster
> started in backup mode. Stalling until master znode is written.
> 2013-05-09 15:08:05,589 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
> already exists and this is not a retry
> 2013-05-09 15:08:05,590 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
> /hbase/backup-masters/xxx.com,6,1368137285373 in backup master directory
> 2013-05-09 15:08:05,595 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the
> active master, xxx.com,6,1368137283107; waiting to become the next active
> master
> 2013-05-09 15:09:45,006 DEBUG
> org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
> Notifying waiting threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
> went down before this master became active
> 2013-05-09 15:09:45,006 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 6
>
> In ActiveMasterManager::blockUntilBecomingActiveMaster()
> {code}
> ..
> if (!clusterStatusTracker.isClusterUp()) {
> this.master.stop(
> "Cluster went down before this master became active");
> }
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8462) Custom timestamps should not be allowed to be negative
[ https://issues.apache.org/jira/browse/HBASE-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8462: - Attachment: hbase-8462_v2.patch Two tests failed because they were using negative time ranges. v2 patch fixes those tests. > Custom timestamps should not be allowed to be negative > -- > > Key: HBASE-8462 > URL: https://issues.apache.org/jira/browse/HBASE-8462 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.98.0, 0.94.8, 0.95.1 > > Attachments: hbase-8462_v1.patch, hbase-8462_v2.patch > > > Client supplied timestamps should not be allowed to be negative, otherwise > unpredictable results will follow. Especially, since we are encoding the ts > using Bytes.Bytes(long), negative timestamps are sorted after positive ones. > Plus, the new PB messages define ts' as uint64. > Credit goes to Huned Lokhandwala for reporting this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8450) Update hbase-default.xml and general recommendations to better suit current hw, h2, experience, etc.
[ https://issues.apache.org/jira/browse/HBASE-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655109#comment-13655109 ] Enis Soztutar commented on HBASE-8450: -- bq. Also are we adding all the configs to xml when it is getting added to code? I doubt this.. I have checked with 94 code base some time back and I was able to find some of the configs only referred in code but not in hbase-default.xml I agree to the concern. I always saw this as a three tiered approach. (1)There are configs which are changed in every hbase-site. (2) There are some configs that you may or may not want to changed. (3) And there are configs that only if you know the code, and something crazy happens, you want to change it. For 1, and 2, they should live in hbase-default. for 3, I agree with Stack, that we should not pollute hbase-default too much. However, I have seen some (2) cases but the config was not in hbase-default. I think we should just fix those cases. > Update hbase-default.xml and general recommendations to better suit current > hw, h2, experience, etc. > > > Key: HBASE-8450 > URL: https://issues.apache.org/jira/browse/HBASE-8450 > Project: HBase > Issue Type: Task > Components: Usability >Reporter: stack >Priority: Critical > Fix For: 0.95.1 > > Attachments: 8450.txt > > > This is a critical task we need to do before we release; review our defaults. > On cursory review, there are configs in hbase-default.xml that no longer have > matching code; there are some that should be changed because we know better > now and others that should be amended because hardware and deploys are bigger > than they used to be. > We could also move stuff around so that the must-edit stuff is near the top > (zk quorum config. is mid-way down the page) and beef up the descriptions -- > especially since these descriptions shine through in the refguide. > Lastly, I notice that our tgz does not "include" an hbase-default.xml other > than the one bundled up in the jar. Maybe we should make it more accessible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8375: - Attachment: hbase-8375_v2.patch v2 from RB. > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.98.0, 0.95.1 > > Attachments: hbase-8375_v1.patch, hbase-8375_v2.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6134) Improvement for split-worker to speed up distributed log splitting
[ https://issues.apache.org/jira/browse/HBASE-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655106#comment-13655106 ] Liang Lee commented on HBASE-6134: -- Hi,when I apply the patch in Hbase-0.94.2 ,I find error in the HLogSplitter.java: The object DistributedLogSplittingHelper can't find its defination,what's the matter? Thanks > Improvement for split-worker to speed up distributed log splitting > -- > > Key: HBASE-6134 > URL: https://issues.apache.org/jira/browse/HBASE-6134 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.95.0 > > Attachments: 6134v4.patch, HBASE-6134.patch, HBASE-6134v2.patch, > HBASE-6134v3-92.patch, HBASE-6134v3.patch, HBASE-6134v4-94.patch, > HBASE-6134v4-94v2.patch, HBASE-6134v4.patch > > > First，we do the test between local-master-splitting and > distributed-log-splitting > Environment：34 hlog files, 5 regionservers,(after kill one, only 4 rs do ths > splitting work), 400 regions in one hlog file > local-master-split:60s+ > distributed-log-splitting:165s+ > In fact, in our production environment, distributed-log-splitting also took > 60s with 30 regionservers for 34 hlog files (regionserver may be in high load) > We found split-worker split one log file took about 20s > (30ms~50ms per writer.close(); 10ms per create writers ) > I think we could do the improvement for this: > Parallelizing the create and close writers in threads > In the patch, change the logic for distributed-log-splitting same as the > local-master-splitting and parallelizing the close in threads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8520) TestIOFencing fails intermittently due to compaction kicking in too soon
[
https://issues.apache.org/jira/browse/HBASE-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655105#comment-13655105
]
Hudson commented on HBASE-8520:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #524 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/524/])
HBASE-8520 TestIOFencing fails intermittently due to compaction kicking in
too soon (Ted Yu) (Revision 1481226)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
> TestIOFencing fails intermittently due to compaction kicking in too soon
>
>
> Key: HBASE-8520
> URL: https://issues.apache.org/jira/browse/HBASE-8520
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0, 0.95.1
>
> Attachments: 8520-v1.txt
>
>
> Here is one example:
> https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase/TestIOFencing/testFencingAroundCompactionAfterWALSync/
> It failed on this assertion:
> {code}
> assertTrue(compactingRegion.countStoreFiles() > 1);
> {code}
> I searched for "Blocking compactions" but didn't find it in the test output.
> I think the compaction kicked in too soon, resulting in store file count
> being 1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655103#comment-13655103
]
Hadoop QA commented on HBASE-8517:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582741/HBASE-8517-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:855)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5634//console
This message is automatically generated.
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were tri
[jira] [Commented] (HBASE-7006) [MTTR] Improve Region Server Recovery Time - Distributed Log Replay
[
https://issues.apache.org/jira/browse/HBASE-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655100#comment-13655100
]
Hadoop QA commented on HBASE-7006:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582743/hbase-7006-combined-v8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 31 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5635//console
This message is automatically generated.
> [MTTR] Improve Region Server Recovery Time - Distributed Log Replay
> ---
>
> Key: HBASE-7006
> URL: https://issues.apache.org/jira/browse/HBASE-7006
> Project: HBase
> Issue Type: New Feature
> Components: MTTR
>Reporter: stack
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.95.1
>
> Attachments: hbase-7006-combined.patch, hbase-7006-combined-v1.patch,
> hbase-7006-combined-v4.patch, hbase-7006-combined-v5.patch,
> hbase-7006-combined-v6.patch, hbase-7006-combined-v7.patch,
> hbase-7006-combined-v8.patch, LogSplitting Comparison.pdf,
> ProposaltoimprovelogsplittingprocessregardingtoHBASE-7006-v2.pdf
>
>
> Just saw interesting issue where a cluster went down hard and 30 nodes had
> 1700 WALs to replay. Replay took almost an hour. It looks like it could run
> faster that much of the time is spent zk'ing and nn'ing.
> Putting in 0.96 so it gets a look at least. Can always punt.
> Distributed Log Replay Description:
> After a region server fails, we firstly assign a failed region to another
> region server with recovering state marked in ZooKeeper. Then a
> SplitLogWorker directly replays edits from WAL(Write-Ahead-Log)s of the
> failed region server to the region after it's re-opened in the new location.
> When a region is in recovering state, it can also accept writes but no
> reads(including Append and Increment), region split or merge.
> The feature piggybacks on existing distributed log splitting framework and
> directly replay WAL edits to another region server instead of creating
> recovered.edits files.
> The advantages over existing log splitting recovered edits implementation:
> 1) Eliminate the steps to write and read recovered.edits files. There could
> be thousands of recovered.edits files are created and written concurrently
> during a region server recovery. Many small random writes could degrade the
>
[jira] [Commented] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655093#comment-13655093
]
Gary Helmling commented on HBASE-8528:
--
bq. I like the yarn.is.minicluster setting approach because it is more direct.
I'm wondering why this isn't the default when every minimrcluster is created.
This is set in the MiniMRCluster configuration automatically, but the
Configuration instance seems to be a copy of any Configuration passed down to
the MiniMRCluster constructor, so it isn't visible externally. At the moment
HBaseTestingUtility doesn't even pass down the Configuration instance, but even
if it did it wouldn't help due to the above.
If you see any more transparent way of getting yarn.is.minicluster set
automatically, that would be cool.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528_alt.patch, hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5995) Fix and reenable TestLogRolling.testLogRollOnPipelineRestart
[
https://issues.apache.org/jira/browse/HBASE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655090#comment-13655090
]
Hadoop QA commented on HBASE-5995:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582735/hbase-5995_v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5633//console
This message is automatically generated.
> Fix and reenable TestLogRolling.testLogRollOnPipelineRestart
>
>
> Key: HBASE-5995
> URL: https://issues.apache.org/jira/browse/HBASE-5995
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: Enis Soztutar
>Priority: Blocker
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-5995_v1.patch, hbase-5995_v2.patch
>
>
> HBASE-5984 disabled this flakey test (See the issue for more). This issue is
> about getting it enabled again. Made a blocker on 0.96.0 so it gets
> attention.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8520) TestIOFencing fails intermittently due to compaction kicking in too soon
[
https://issues.apache.org/jira/browse/HBASE-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655087#comment-13655087
]
Hudson commented on HBASE-8520:
---
Integrated in hbase-0.95 #186 (See
[https://builds.apache.org/job/hbase-0.95/186/])
HBASE-8520 TestIOFencing fails intermittently due to compaction kicking in
too soon (Ted Yu) (Revision 1481227)
Result = SUCCESS
tedyu :
Files :
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
> TestIOFencing fails intermittently due to compaction kicking in too soon
>
>
> Key: HBASE-8520
> URL: https://issues.apache.org/jira/browse/HBASE-8520
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0, 0.95.1
>
> Attachments: 8520-v1.txt
>
>
> Here is one example:
> https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase/TestIOFencing/testFencingAroundCompactionAfterWALSync/
> It failed on this assertion:
> {code}
> assertTrue(compactingRegion.countStoreFiles() > 1);
> {code}
> I searched for "Blocking compactions" but didn't find it in the test output.
> I think the compaction kicked in too soon, resulting in store file count
> being 1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8520) TestIOFencing fails intermittently due to compaction kicking in too soon
[
https://issues.apache.org/jira/browse/HBASE-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655086#comment-13655086
]
Hudson commented on HBASE-8520:
---
Integrated in HBase-TRUNK #4110 (See
[https://builds.apache.org/job/HBase-TRUNK/4110/])
HBASE-8520 TestIOFencing fails intermittently due to compaction kicking in
too soon (Ted Yu) (Revision 1481226)
Result = SUCCESS
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
> TestIOFencing fails intermittently due to compaction kicking in too soon
>
>
> Key: HBASE-8520
> URL: https://issues.apache.org/jira/browse/HBASE-8520
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0, 0.95.1
>
> Attachments: 8520-v1.txt
>
>
> Here is one example:
> https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase/TestIOFencing/testFencingAroundCompactionAfterWALSync/
> It failed on this assertion:
> {code}
> assertTrue(compactingRegion.countStoreFiles() > 1);
> {code}
> I searched for "Blocking compactions" but didn't find it in the test output.
> I think the compaction kicked in too soon, resulting in store file count
> being 1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655083#comment-13655083 ] Roshan Naik commented on HBASE-6929: these are not hadoop2 specific these are hbase0.95+ specific which comes in both hadoop1&2 flovaours. So it seems we will need to support the following combos... 1) building against hbase 0.94x .. in this case there will be no -hadoop1 suffix on hbase deps 2) building against hbase 0.95+ with hadoop 1 .. in this case the -hadoop1 suffix will be used with the newer dependencies list 3) building against hbase 0.95+ with hadoop 2 .. in this case the -hadoop2 suffix will be used with the newer dependency list > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8520) TestIOFencing fails intermittently due to compaction kicking in too soon
[
https://issues.apache.org/jira/browse/HBASE-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655082#comment-13655082
]
Hudson commented on HBASE-8520:
---
Integrated in hbase-0.95-on-hadoop2 #94 (See
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/94/])
HBASE-8520 TestIOFencing fails intermittently due to compaction kicking in
too soon (Ted Yu) (Revision 1481227)
Result = FAILURE
tedyu :
Files :
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
> TestIOFencing fails intermittently due to compaction kicking in too soon
>
>
> Key: HBASE-8520
> URL: https://issues.apache.org/jira/browse/HBASE-8520
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0, 0.95.1
>
> Attachments: 8520-v1.txt
>
>
> Here is one example:
> https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase/TestIOFencing/testFencingAroundCompactionAfterWALSync/
> It failed on this assertion:
> {code}
> assertTrue(compactingRegion.countStoreFiles() > 1);
> {code}
> I searched for "Blocking compactions" but didn't find it in the test output.
> I think the compaction kicked in too soon, resulting in store file count
> being 1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655079#comment-13655079 ] Hari Shreedharan commented on HBASE-6929: - Yep, we could define those in the hadoop-2 profile as dependencies. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655078#comment-13655078 ] Enis Soztutar commented on HBASE-8375: -- Wow, totally forgot that there is a TestDurability class. Let me fix the test. > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.98.0, 0.95.1 > > Attachments: hbase-8375_v1.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8375) Streamline Table durability settings
[
https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655075#comment-13655075
]
Hadoop QA commented on HBASE-8375:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582727/hbase-8375_v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestDurability
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5632//console
This message is automatically generated.
> Streamline Table durability settings
>
>
> Key: HBASE-8375
> URL: https://issues.apache.org/jira/browse/HBASE-8375
> Project: HBase
> Issue Type: Sub-task
>Reporter: Lars Hofhansl
>Assignee: Enis Soztutar
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8375_v1.patch
>
>
> HBASE-7801 introduces the notion of per mutation fine grained durability
> settings.
> This issue is to consider and the discuss the same for the per table settings
> (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose
> the following setting per table:
> * SKIP_WAL (i.e. an unlogged table)
> * ASYNC_WAL (the current deferred log flush)
> * SYNC_WAL (the current default)
> * FSYNC_WAL (for future uses of HDFS' hsync())
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655074#comment-13655074
]
Hadoop QA commented on HBASE-8528:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582734/hbase-8528_alt.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5631//console
This message is automatically generated.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528_alt.patch, hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655071#comment-13655071
]
Ted Yu commented on HBASE-8528:
---
+1 on Gary's patch.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528_alt.patch, hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655067#comment-13655067
]
Elliott Clark edited comment on HBASE-8517 at 5/11/13 12:23 AM:
bq.How was the constant 600 determined ?
That was the default before adding the scaling.
bq.Should it be related to cluster size (number of servers, number of regions) ?
Possibly but this works well, and number of servers should on a well run
cluster be a proxy for number of regions; so it's not really needed in the
computation.
bq.Looks like the return value from computeCost() is ignored.
Yes it is just used to output a good debug message.
bq.Is the addition of 9 needed when cluster.numMovedMetaRegions is 0 ?
Yes, because the possibility is there. You want to scale based upon the
highest possible cost.
was (Author: eclark):
bq.How was the constant 600 determined ?
That was the default before adding the scaling.
bq.Should it be related to cluster size (number of servers, number of regions) ?
Possibly but this works well, and number of servers should on a well run
cluster be a proxy for number of regions.
bq.Looks like the return value from computeCost() is ignored.
Yes it is just used to output a good debug message.
bq.Is the addition of 9 needed when cluster.numMovedMetaRegions is 0 ?
Yes, because the possibility is there. You want to scale based upon the
highest possible cost.
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655067#comment-13655067
]
Elliott Clark commented on HBASE-8517:
--
bq.How was the constant 600 determined ?
That was the default before adding the scaling.
bq.Should it be related to cluster size (number of servers, number of regions) ?
Possibly but this works well, and number of servers should on a well run
cluster be a proxy for number of regions.
bq.Looks like the return value from computeCost() is ignored.
Yes it is just used to output a good debug message.
bq.Is the addition of 9 needed when cluster.numMovedMetaRegions is 0 ?
Yes, because the possibility is there. You want to scale based upon the
highest possible cost.
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655066#comment-13655066 ] Roshan Naik commented on HBASE-6929: Hari: it looks like flume needs these .. hbase-client, hbase-common (HBaseCommonTestingUtility.. at compile time), hbase-server (HMaster,MiniZooKeeperCluster.. at compile time) and hbase-server-test (HBaseTestingUtility..when running tests) > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8527) clean up code around compaction completion in HStore
[
https://issues.apache.org/jira/browse/HBASE-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655065#comment-13655065
]
Ted Yu commented on HBASE-8527:
---
{code}
- } finally {
-// We need the lock, as long as we are updating the storeFiles
-// or changing the memstore. Let us release it before calling
-// notifyChangeReadersObservers. See HBASE-4485 for a possible
-// deadlock scenario that could have happened if continue to hold
-// the lock.
-this.lock.writeLock().unlock();
{code}
I see the this.lock.writeLock().unlock() call is moved. Better move the comment
as well.
> clean up code around compaction completion in HStore
>
>
> Key: HBASE-8527
> URL: https://issues.apache.org/jira/browse/HBASE-8527
> Project: HBase
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-8527-v0.patch
>
>
> The methods completeCompaction and it's caller are too long. Something I
> changed while doing something else there; putting in a separate
> easy-to-review JIRA to make the other future change smaller.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8529) checkOpen is missing from multi, mutate, get and multiGet etc.
Jeffrey Zhong created HBASE-8529: Summary: checkOpen is missing from multi, mutate, get and multiGet etc. Key: HBASE-8529 URL: https://issues.apache.org/jira/browse/HBASE-8529 Project: HBase Issue Type: Bug Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Priority: Minor Fix For: 0.98.0, 0.95.1 I saw we have checkOpen in all those functions in 0.94 while they're missing from trunk. Does anyone know why? For multi and mutate, if we don't call checkOpen we could flood our logs with bunch of "DFSOutputStream is closed" errors when we sync WAL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655063#comment-13655063
]
Jonathan Hsieh commented on HBASE-8528:
---
[~ghelmling], I like the {{yarn.is.minicluster}} setting approach because it is
more direct. I'm wondering why this isn't the default when every minimrcluster
is created.
I brought this up with our mr2 guy, [~sandyr] -- he's going to do a quick look
and post here explaining why.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528_alt.patch, hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655053#comment-13655053
]
Ted Yu commented on HBASE-8517:
---
{code}
+maxMoves = Math.max( (int) (cluster.numRegions * maxMovesPercent),
+600);
{code}
How was the constant 600 determined ?
Should it be related to cluster size (number of servers, number of regions) ?
{code}
if (initCost > currentCost) {
+ computeCost(cluster, true);
{code}
Looks like the return value from computeCost() is ignored.
{code}
+return scale(0, cluster.numRegions+9, moveCost);
{code}
Is the addition of 9 needed when cluster.numMovedMetaRegions is 0 ?
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[
https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655051#comment-13655051
]
Hadoop QA commented on HBASE-8154:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582720/8154-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5630//console
This message is automatically generated.
> HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address
> already in use" and retry automatically
> -
>
> Key: HBASE-8154
> URL: https://issues.apache.org/jira/browse/HBASE-8154
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 0.98.0
>Reporter: Liang Xie
>Assignee: Ted Yu
> Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/
> and
> https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8519) Backup master will never come up if primary master dies during initialization
[
https://issues.apache.org/jira/browse/HBASE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655047#comment-13655047
]
Jean-Daniel Cryans commented on HBASE-8519:
---
bq. I think Jerry's description mentioned that
clusterStatusTracker.isClusterUp() returned false, leading to backup master
stopping.
Sorry, I meant the reverse for the shutdown znode. Which should also answer
Jerry's second question.
bq. Can you give more details on the sequence of events?
Do a stop-hbase then kill the master just after that, you should get the same
"Cluster went down before this master became active" when you restart.
> Backup master will never come up if primary master dies during initialization
> -
>
> Key: HBASE-8519
> URL: https://issues.apache.org/jira/browse/HBASE-8519
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.7, 0.95.0
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.98.0
>
>
> The problem happens if primary master dies after becoming master but before
> it completes initialization and calls clusterStatusTracker.setClusterUp(),
> The backup master will try to become the master, but will shutdown itself
> promptly because it sees 'the cluster is not up'.
> This is the backup master log:
> 2013-05-09 15:08:05,568 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2013-05-09 15:08:05,573 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster
> started in backup mode. Stalling until master znode is written.
> 2013-05-09 15:08:05,589 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
> already exists and this is not a retry
> 2013-05-09 15:08:05,590 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
> /hbase/backup-masters/xxx.com,6,1368137285373 in backup master directory
> 2013-05-09 15:08:05,595 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the
> active master, xxx.com,6,1368137283107; waiting to become the next active
> master
> 2013-05-09 15:09:45,006 DEBUG
> org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
> Notifying waiting threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
> went down before this master became active
> 2013-05-09 15:09:45,006 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 6
>
> In ActiveMasterManager::blockUntilBecomingActiveMaster()
> {code}
> ..
> if (!clusterStatusTracker.isClusterUp()) {
> this.master.stop(
> "Cluster went down before this master became active");
> }
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7006) [MTTR] Improve Region Server Recovery Time - Distributed Log Replay
[ https://issues.apache.org/jira/browse/HBASE-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-7006: - Attachment: hbase-7006-combined-v8.patch V8 patch: Rebased and one line change add "checkOpen()" into replay command(I found "multi, mutate and multiGet" all missing checkOpen call. I'll file a separate JIRA on this). Summary on the JIRA: All review comments are addressed. Unit tests passed and no issue is found so far from integration test IntegrationTestDataIngestWithChaosMonkey(will keeping running for a while). I think we can check it in now. Next steps are: 1) Re-run performance tests to avoid any performance degradation from recent changes. 2) Add configuration to "disallow writes during recovery", which I intentionally leave it out from the combined patch. Thanks. > [MTTR] Improve Region Server Recovery Time - Distributed Log Replay > --- > > Key: HBASE-7006 > URL: https://issues.apache.org/jira/browse/HBASE-7006 > Project: HBase > Issue Type: New Feature > Components: MTTR >Reporter: stack >Assignee: Jeffrey Zhong >Priority: Critical > Fix For: 0.95.1 > > Attachments: hbase-7006-combined.patch, hbase-7006-combined-v1.patch, > hbase-7006-combined-v4.patch, hbase-7006-combined-v5.patch, > hbase-7006-combined-v6.patch, hbase-7006-combined-v7.patch, > hbase-7006-combined-v8.patch, LogSplitting Comparison.pdf, > ProposaltoimprovelogsplittingprocessregardingtoHBASE-7006-v2.pdf > > > Just saw interesting issue where a cluster went down hard and 30 nodes had > 1700 WALs to replay. Replay took almost an hour. It looks like it could run > faster that much of the time is spent zk'ing and nn'ing. > Putting in 0.96 so it gets a look at least. Can always punt. > Distributed Log Replay Description: > After a region server fails, we firstly assign a failed region to another > region server with recovering state marked in ZooKeeper. Then a > SplitLogWorker directly replays edits from WAL(Write-Ahead-Log)s of the > failed region server to the region after it's re-opened in the new location. > When a region is in recovering state, it can also accept writes but no > reads(including Append and Increment), region split or merge. > The feature piggybacks on existing distributed log splitting framework and > directly replay WAL edits to another region server instead of creating > recovered.edits files. > The advantages over existing log splitting recovered edits implementation: > 1) Eliminate the steps to write and read recovered.edits files. There could > be thousands of recovered.edits files are created and written concurrently > during a region server recovery. Many small random writes could degrade the > overall system performance. > 2) Allow writes even when a region is in recovering state. It only takes > seconds for a failed over region to accept writes again. > The feature can be disabled by setting hbase.master.distributed.log.replay to > false(by default is true) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8519) Backup master will never come up if primary master dies during initialization
[
https://issues.apache.org/jira/browse/HBASE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655040#comment-13655040
]
Jerry He commented on HBASE-8519:
-
bq. If you kill the master while it's shutting down the cluster and you try to
restart HBase, it will have the same error and will go down (and the cluster
won't even come up).
Can you give more details on the sequence of events?
bq. All you need is the master and the shutdown znodes in place when starting
HMaster.
If the shutdown/clusterStateZNode znode exists, we will be ok thru this code,
correct?
{code}
..
if (!clusterStatusTracker.isClusterUp()) {
this.master.stop(
"Cluster went down before this master became active");
}
..
{code}
The master znode is ephemeral, and the clusterStateZNode is persistent.
> Backup master will never come up if primary master dies during initialization
> -
>
> Key: HBASE-8519
> URL: https://issues.apache.org/jira/browse/HBASE-8519
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.7, 0.95.0
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.98.0
>
>
> The problem happens if primary master dies after becoming master but before
> it completes initialization and calls clusterStatusTracker.setClusterUp(),
> The backup master will try to become the master, but will shutdown itself
> promptly because it sees 'the cluster is not up'.
> This is the backup master log:
> 2013-05-09 15:08:05,568 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2013-05-09 15:08:05,573 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster
> started in backup mode. Stalling until master znode is written.
> 2013-05-09 15:08:05,589 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
> already exists and this is not a retry
> 2013-05-09 15:08:05,590 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
> /hbase/backup-masters/xxx.com,6,1368137285373 in backup master directory
> 2013-05-09 15:08:05,595 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the
> active master, xxx.com,6,1368137283107; waiting to become the next active
> master
> 2013-05-09 15:09:45,006 DEBUG
> org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
> Notifying waiting threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
> went down before this master became active
> 2013-05-09 15:09:45,006 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 6
>
> In ActiveMasterManager::blockUntilBecomingActiveMaster()
> {code}
> ..
> if (!clusterStatusTracker.isClusterUp()) {
> this.master.stop(
> "Cluster went down before this master became active");
> }
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-8517:
-
Attachment: HBASE-8517-1.patch
Without the long line.
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch, HBASE-8517-1.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8519) Backup master will never come up if primary master dies during initialization
[
https://issues.apache.org/jira/browse/HBASE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655032#comment-13655032
]
Ted Yu commented on HBASE-8519:
---
bq. Finally, this jira, you have both
I think Jerry's description mentioned that clusterStatusTracker.isClusterUp()
returned false, leading to backup master stopping.
> Backup master will never come up if primary master dies during initialization
> -
>
> Key: HBASE-8519
> URL: https://issues.apache.org/jira/browse/HBASE-8519
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.7, 0.95.0
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.98.0
>
>
> The problem happens if primary master dies after becoming master but before
> it completes initialization and calls clusterStatusTracker.setClusterUp(),
> The backup master will try to become the master, but will shutdown itself
> promptly because it sees 'the cluster is not up'.
> This is the backup master log:
> 2013-05-09 15:08:05,568 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2013-05-09 15:08:05,573 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster
> started in backup mode. Stalling until master znode is written.
> 2013-05-09 15:08:05,589 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
> already exists and this is not a retry
> 2013-05-09 15:08:05,590 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
> /hbase/backup-masters/xxx.com,6,1368137285373 in backup master directory
> 2013-05-09 15:08:05,595 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the
> active master, xxx.com,6,1368137283107; waiting to become the next active
> master
> 2013-05-09 15:09:45,006 DEBUG
> org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
> Notifying waiting threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
> went down before this master became active
> 2013-05-09 15:09:45,006 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 6
>
> In ActiveMasterManager::blockUntilBecomingActiveMaster()
> {code}
> ..
> if (!clusterStatusTracker.isClusterUp()) {
> this.master.stop(
> "Cluster went down before this master became active");
> }
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655030#comment-13655030 ] Jimmy Xiang commented on HBASE-8420: The configurable max prefetched result size is set to 512 M by default. Whenever a scanner is called to fetch data, we will check if the caller thread is a prefetcher thread based on the thread name. If it is a prefetcher thread, the result wire size is added to global atomic long variable (a counter). After the prefetched result is retrieved, the same amount is reduced from the counter. If the counter value is over the pre-configured threshold, the next prefetching will not be scheduled. This is a rough control. We don't do micro control, i.e. stop a prefetcher thread in the middle. > Port HBASE-6874 Implement prefetching for scanners from 0.89-fb > - > > Key: HBASE-8420 > URL: https://issues.apache.org/jira/browse/HBASE-8420 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, > trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch > > > This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655026#comment-13655026 ] Enis Soztutar commented on HBASE-6929: -- I think Flume needs at least the hbase-server tests jar to use the minicluster. We currently do not have hbase-minicluster kind of module. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8527) clean up code around compaction completion in HStore
[
https://issues.apache.org/jira/browse/HBASE-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655020#comment-13655020
]
Hadoop QA commented on HBASE-8527:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582707/HBASE-8527-v0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5628//console
This message is automatically generated.
> clean up code around compaction completion in HStore
>
>
> Key: HBASE-8527
> URL: https://issues.apache.org/jira/browse/HBASE-8527
> Project: HBase
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-8527-v0.patch
>
>
> The methods completeCompaction and it's caller are too long. Something I
> changed while doing something else there; putting in a separate
> easy-to-review JIRA to make the other future change smaller.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655017#comment-13655017
]
Hadoop QA commented on HBASE-8517:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582702/HBASE-8517-0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5629//console
This message is automatically generated.
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more infor
[jira] [Updated] (HBASE-5995) Fix and reenable TestLogRolling.testLogRollOnPipelineRestart
[ https://issues.apache.org/jira/browse/HBASE-5995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-5995: - Attachment: hbase-5995_v2.patch Here is a second attempt. This patch also fixes a condition where we the hdfs output stream is closed because of errors, but we try to continuously do hflush() before closing the stream, although it is already closed. FSDataOutputStream does not have an isClosed() kind of API, or throw a special exception, so I had to parse the exception msg (which is ugly I admit). In hadoop2, we check the GS's of the block and replica to reason about the file length, which is why we get the error. In hadoop1, it seems that we do not do that, and except the replica length with checking GS (I might be wrong on this though, haven't confirmed it with hdfs folks). > Fix and reenable TestLogRolling.testLogRollOnPipelineRestart > > > Key: HBASE-5995 > URL: https://issues.apache.org/jira/browse/HBASE-5995 > Project: HBase > Issue Type: Sub-task > Components: test >Reporter: stack >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 0.98.0, 0.95.1 > > Attachments: hbase-5995_v1.patch, hbase-5995_v2.patch > > > HBASE-5984 disabled this flakey test (See the issue for more). This issue is > about getting it enabled again. Made a blocker on 0.96.0 so it gets > attention. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655007#comment-13655007 ] Hari Shreedharan commented on HBASE-6929: - Roshan: we should not need anything more than the HBase client jars (I believe - not sure where the minicluster classes for tests are and what we'd need to handle that though). So we should have to do only what we do hadoop-common v/s hadoop-core. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-8528:
-
Attachment: hbase-8528_alt.patch
Here's an alternate patch for trunk that sets yarn.is.minicluster.
TestMultiTableInputFormat passes for me with this change.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528_alt.patch, hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654998#comment-13654998 ] Roshan Naik commented on HBASE-6929: there will be a little complexity .. Hbase jar has been split up in 0.95.. so in addition to the hadoop2 profile flume (and others such prjs) will now additionally need a profile for hbase0.95. the complexity is that neither the hadoop2 profile nor the hbase0.95+ profile can independently conjure up the full name and dependencies list for hbase. let me give it a try.. will get back. > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654997#comment-13654997
]
Gary Helmling commented on HBASE-8528:
--
Coincidentally, I've just been looking at the exact same problem when running
against 2.0.4-alpha. Setting {{yarn.is.minicluster}} to "true" in the
configuration solves the problem and in effect does the same as your patch (at
least in 2.0.4-alpha).
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-8528:
--
Status: Patch Available (was: Open)
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8375: - Release Note: Added Durability setting for table with HTableDescriptor.setDurability() API. Deprecated HTableDescriptor.setDeferredLogFlush(), and shell usage in favor of the new durability setting. Now, durability can be configured per table and per mutation. > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.98.0, 0.95.1 > > Attachments: hbase-8375_v1.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654983#comment-13654983
]
Jonathan Hsieh commented on HBASE-8528:
---
Somehow the simplification of the yarn classpaths in YARN-129 doesn't include
as much as before. So we add unit tests's classpath into to the
yarn.application.classpath so that the spawned children have the MR2 jars
necessary to run.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-8528:
--
Issue Type: Sub-task (was: Bug)
Parent: HBASE-6891
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8375: - Status: Patch Available (was: Open) > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.95.2 > > Attachments: hbase-8375_v1.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8375: - Fix Version/s: (was: 0.95.2) 0.95.1 0.98.0 > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.98.0, 0.95.1 > > Attachments: hbase-8375_v1.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654972#comment-13654972 ] Enis Soztutar commented on HBASE-8375: -- Put up a patch at https://reviews.apache.org/r/11064/. Reviews welcome, no rush :) The patch does what the jira description talks about. > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.95.2 > > Attachments: hbase-8375_v1.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8375) Streamline Table durability settings
[ https://issues.apache.org/jira/browse/HBASE-8375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-8375: - Attachment: hbase-8375_v1.patch > Streamline Table durability settings > > > Key: HBASE-8375 > URL: https://issues.apache.org/jira/browse/HBASE-8375 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Enis Soztutar > Fix For: 0.95.2 > > Attachments: hbase-8375_v1.patch > > > HBASE-7801 introduces the notion of per mutation fine grained durability > settings. > This issue is to consider and the discuss the same for the per table settings > (i.e. what would be used if the mutation indicates USE_DEFAULT). I propose > the following setting per table: > * SKIP_WAL (i.e. an unlogged table) > * ASYNC_WAL (the current deferred log flush) > * SYNC_WAL (the current default) > * FSYNC_WAL (for future uses of HDFS' hsync()) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-8528:
--
Attachment: hbase-8528.patch
Attached patch fixes problem for TestMultiTableInputFormat. Waiting for QA bot
to see if it affects and other tests.
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Bug
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
> Attachments: hbase-8528.patch
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
[
https://issues.apache.org/jira/browse/HBASE-8528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-8528:
--
Description:
TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
message isn't useful (assertion failed) -- but if you look at the
stderr/stdout/syslog files of the jobs in
hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
dirs) you'll find this class not found exception:
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
Program will exit.
{code}
was:
TestMulitTableInputFormat always fails. Its error message isn't useful -- but
if you look at the stderr/stdout/syslog files of the jobs in
hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
dirs) you'll find this class not found exception:
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
Program will exit.
{code}
> [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129
> applied
>
>
> Key: HBASE-8528
> URL: https://issues.apache.org/jira/browse/HBASE-8528
> Project: HBase
> Issue Type: Bug
> Components: hadoop2, mapreduce, test
>Affects Versions: 0.95.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 0.98.0, 0.95.1
>
>
> TestMulitTableInputFormat always fails if YARN-129 is applied. Its error
> message isn't useful (assertion failed) -- but if you look at the
> stderr/stdout/syslog files of the jobs in
> hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
> dirs) you'll find this class not found exception:
> {code}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/mapreduce/v2/app/MRAppMaster
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> Could not find the main class:
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit.
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8528) [hadoop2] TestMultiTableInputFormat always fails on hadoop with YARN-129 applied
Jonathan Hsieh created HBASE-8528:
-
Summary: [hadoop2] TestMultiTableInputFormat always fails on
hadoop with YARN-129 applied
Key: HBASE-8528
URL: https://issues.apache.org/jira/browse/HBASE-8528
Project: HBase
Issue Type: Bug
Components: hadoop2, mapreduce, test
Affects Versions: 0.95.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.98.0, 0.95.1
TestMulitTableInputFormat always fails. Its error message isn't useful -- but
if you look at the stderr/stdout/syslog files of the jobs in
hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster* (minicluster data
dirs) you'll find this class not found exception:
{code}
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/mapreduce/v2/app/MRAppMaster
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapreduce.v2.app.MRAppMaster
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster.
Program will exit.
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8483) HConnectionManager can leak ZooKeeper connections when using deleteStaleConnection
[
https://issues.apache.org/jira/browse/HBASE-8483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654964#comment-13654964
]
Ted Yu commented on HBASE-8483:
---
In
https://builds.apache.org/job/HBase-TRUNK/4109/testReport/org.apache.hadoop.hbase.client/TestHCM/testDeleteForZKConnLeak/
:
{code}
java.lang.AssertionError: Live zookeeper in closed connection
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:851)
{code}
> HConnectionManager can leak ZooKeeper connections when using
> deleteStaleConnection
> --
>
> Key: HBASE-8483
> URL: https://issues.apache.org/jira/browse/HBASE-8483
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.94.4
>Reporter: Eric Yu
>Priority: Critical
> Fix For: 0.98.0, 0.94.8, 0.95.1
>
> Attachments: HBASE-8483-0.94.patch, HBASE-8483.patch,
> LeakZKConnections.java
>
>
> If one thread calls deleteStaleConnection while other threads are using
> connection, can leak ZK connections.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[ https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8154: -- Attachment: 8154-v4.txt > HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address > already in use" and retry automatically > - > > Key: HBASE-8154 > URL: https://issues.apache.org/jira/browse/HBASE-8154 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Ted Yu > Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt > > > see > https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ > and > https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[ https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8154: -- Attachment: (was: 8154-v4.txt) > HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address > already in use" and retry automatically > - > > Key: HBASE-8154 > URL: https://issues.apache.org/jira/browse/HBASE-8154 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Ted Yu > Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt > > > see > https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ > and > https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[ https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8154: -- Attachment: (was: 8154-v4.txt) > HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address > already in use" and retry automatically > - > > Key: HBASE-8154 > URL: https://issues.apache.org/jira/browse/HBASE-8154 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Ted Yu > Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt > > > see > https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ > and > https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8520) TestIOFencing fails intermittently due to compaction kicking in too soon
[
https://issues.apache.org/jira/browse/HBASE-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654955#comment-13654955
]
Ted Yu commented on HBASE-8520:
---
Integrated to 0.95 and trunk.
> TestIOFencing fails intermittently due to compaction kicking in too soon
>
>
> Key: HBASE-8520
> URL: https://issues.apache.org/jira/browse/HBASE-8520
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.0, 0.95.1
>
> Attachments: 8520-v1.txt
>
>
> Here is one example:
> https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase/TestIOFencing/testFencingAroundCompactionAfterWALSync/
> It failed on this assertion:
> {code}
> assertTrue(compactingRegion.countStoreFiles() > 1);
> {code}
> I searched for "Blocking compactions" but didn't find it in the test output.
> I think the compaction kicked in too soon, resulting in store file count
> being 1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[
https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654937#comment-13654937
]
Hadoop QA commented on HBASE-8420:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582686/trunk-8420_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.client.TestHCM.testDeleteForZKConnLeak(TestHCM.java:837)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5626//console
This message is automatically generated.
> Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
> -
>
> Key: HBASE-8420
> URL: https://issues.apache.org/jira/browse/HBASE-8420
> Project: HBase
> Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch,
> trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch
>
>
> This should help scanner performance. We should have it in trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654934#comment-13654934 ] Hari Shreedharan commented on HBASE-6929: - [~saint@gmail.com], This should work for Flume. We have different mvn profiles for hadoop-1 and hadoop-2. So we can just change the hadoop-2 profile to pull in the correct artifact (we already do this for hadoop-common/hadoop-core). > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-8517:
-
Affects Version/s: 0.95.0
Status: Patch Available (was: Open)
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.95.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[
https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654927#comment-13654927
]
Hadoop QA commented on HBASE-8154:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582688/8154-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5627//console
This message is automatically generated.
> HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address
> already in use" and retry automatically
> -
>
> Key: HBASE-8154
> URL: https://issues.apache.org/jira/browse/HBASE-8154
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 0.98.0
>Reporter: Liang Xie
>Assignee: Ted Yu
> Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt, 8154-v4.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/
> and
> https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8527) clean up code around compaction completion in HStore
[ https://issues.apache.org/jira/browse/HBASE-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8527: Status: Patch Available (was: Open) > clean up code around compaction completion in HStore > > > Key: HBASE-8527 > URL: https://issues.apache.org/jira/browse/HBASE-8527 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-8527-v0.patch > > > The methods completeCompaction and it's caller are too long. Something I > changed while doing something else there; putting in a separate > easy-to-review JIRA to make the other future change smaller. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8527) clean up code around compaction completion in HStore
[ https://issues.apache.org/jira/browse/HBASE-8527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-8527: Attachment: HBASE-8527-v0.patch Move some stuff into methods, mostly > clean up code around compaction completion in HStore > > > Key: HBASE-8527 > URL: https://issues.apache.org/jira/browse/HBASE-8527 > Project: HBase > Issue Type: Improvement >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-8527-v0.patch > > > The methods completeCompaction and it's caller are too long. Something I > changed while doing something else there; putting in a separate > easy-to-review JIRA to make the other future change smaller. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8517) Stochastic Loadbalancer isn't finding steady state on real clusters
[
https://issues.apache.org/jira/browse/HBASE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark updated HBASE-8517:
-
Attachment: HBASE-8517-0.patch
Here's a patch that solves the issue I was seeing on a cluster.
Most of the issue was upping the move cost so that all cost functions below it
couldn't overcome the cost of a move. In other words the balancer will find
the best moves when the region load is our of wack, but it won't move just to
make locality better.
Then I also took a crack at making this thing work on larger clusters.
To do that I made it closer to simulated annealing. It picks either a random
server, or the most loaded server. This means that more often we have a chance
to work on the worst balanced server.
Then I also made the algorithm prefer to move regions off of the over loaded
servers.
Then I made the max number of moves scale with cluster size.
> Stochastic Loadbalancer isn't finding steady state on real clusters
> ---
>
> Key: HBASE-8517
> URL: https://issues.apache.org/jira/browse/HBASE-8517
> Project: HBase
> Issue Type: Bug
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-8517-0.patch
>
>
> I have a cluster that runs IT tests. Last night after all tests were done I
> noticed that the balancer was thrashing regions around.
> The number of regions on each machine is not correct.
> The balancer seems to value the cost of moving a region way too little.
> {code}
> 2013-05-09 16:34:58,920 DEBUG [IPC Server handler 4 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 5367ms to try 8910
> different iterations. Found a solution that moves 37 regions; Going from a
> computed cost of 56.50254222730425 to a new cost of 11.214035466575254
> 2013-05-09 16:37:48,715 DEBUG [IPC Server handler 7 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4735ms to try 8910
> different iterations. Found a solution that moves 38 regions; Going from a
> computed cost of 56.612624531830996 to a new cost of 11.275763861636982
> 2013-05-09 16:38:11,398 DEBUG [IPC Server handler 6 on 6]
> org.apache.hadoop.hbase.master.balancer.StochasticLoadBalancer: Finished
> computing new load balance plan. Computation took 4502ms to try 8910
> different iterations. Found a solution that moves 39 regions; Going from a
> computed cost of 56.50048461413552 to a new cost of 11.225352339003237
> {code}
> Each of those balancer runs were triggered when there was no load on the
> cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7006) [MTTR] Improve Region Server Recovery Time - Distributed Log Replay
[ https://issues.apache.org/jira/browse/HBASE-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654874#comment-13654874 ] stack commented on HBASE-7006: -- Where are we w/ this [~jeffreyz]? What is your sense? Can you summarize. Are all the reviews up in rb addressed in this v7? > [MTTR] Improve Region Server Recovery Time - Distributed Log Replay > --- > > Key: HBASE-7006 > URL: https://issues.apache.org/jira/browse/HBASE-7006 > Project: HBase > Issue Type: New Feature > Components: MTTR >Reporter: stack >Assignee: Jeffrey Zhong >Priority: Critical > Fix For: 0.95.1 > > Attachments: hbase-7006-combined.patch, hbase-7006-combined-v1.patch, > hbase-7006-combined-v4.patch, hbase-7006-combined-v5.patch, > hbase-7006-combined-v6.patch, hbase-7006-combined-v7.patch, LogSplitting > Comparison.pdf, > ProposaltoimprovelogsplittingprocessregardingtoHBASE-7006-v2.pdf > > > Just saw interesting issue where a cluster went down hard and 30 nodes had > 1700 WALs to replay. Replay took almost an hour. It looks like it could run > faster that much of the time is spent zk'ing and nn'ing. > Putting in 0.96 so it gets a look at least. Can always punt. > Distributed Log Replay Description: > After a region server fails, we firstly assign a failed region to another > region server with recovering state marked in ZooKeeper. Then a > SplitLogWorker directly replays edits from WAL(Write-Ahead-Log)s of the > failed region server to the region after it's re-opened in the new location. > When a region is in recovering state, it can also accept writes but no > reads(including Append and Increment), region split or merge. > The feature piggybacks on existing distributed log splitting framework and > directly replay WAL edits to another region server instead of creating > recovered.edits files. > The advantages over existing log splitting recovered edits implementation: > 1) Eliminate the steps to write and read recovered.edits files. There could > be thousands of recovered.edits files are created and written concurrently > during a region server recovery. Many small random writes could degrade the > overall system performance. > 2) Allow writes even when a region is in recovering state. It only takes > seconds for a failed over region to accept writes again. > The feature can be disabled by setting hbase.master.distributed.log.replay to > false(by default is true) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654870#comment-13654870 ] stack commented on HBASE-6929: -- [~roshan_naik] Here is our attempt: https://repository.apache.org/content/groups/snapshots/org/apache/hbase/hbase-client/ What do you think? Will it work for you? We publish two artifacts, one named w/ a hadoop1 suffix and the other with a hadoop2. Would be interested in feedback. What I have so far is that this scheme will not work (our Elliott has it our hadoop dependency is resolved before we have a chance to influence it -- profiles). > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654866#comment-13654866 ] Jimmy Xiang edited comment on HBASE-8420 at 5/10/13 9:30 PM: - Did some quick scan testing with YCSB on a same cluster: 1 master node + 4 region server nodes. All are the same except the hbase jar file. Total rows: 3,000,000. Scan ops count: 3,000 (not that many). With prefetching (v2): throughput: 36.9 ops/sec, avg latency: 27.0 ms, 95th: 48 ms, 99th: 69 ms Without prefetching (without the patch at all): throughput: 32.8 ops/sec, avg latency: 30.4 ms, 95th: 50 ms, 99th: 227 ms On average, the latency is about 10% shorter. was (Author: jxiang): Did some quest scan testing with YCSB on a same cluster: 1 master node + 4 region server nodes. All are the same except the hbase jar file. Total rows: 3,000,000. Scan ops count: 3,000 (not that many). With prefetching (v2): throughput: 36.9 ops/sec, avg latency: 27.0 ms, 95th: 48 ms, 99th: 69 ms Without prefetching (without the patch at all): throughput: 32.8 ops/sec, avg latency: 30.4 ms, 95th: 50 ms, 99th: 227 ms On average, the latency is about 10% shorter. > Port HBASE-6874 Implement prefetching for scanners from 0.89-fb > - > > Key: HBASE-8420 > URL: https://issues.apache.org/jira/browse/HBASE-8420 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, > trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch > > > This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654866#comment-13654866 ] Jimmy Xiang commented on HBASE-8420: Did some quest scan testing with YCSB on a same cluster: 1 master node + 4 region server nodes. All are the same except the hbase jar file. Total rows: 3,000,000. Scan ops count: 3,000 (not that many). With prefetching (v2): throughput: 36.9 ops/sec, avg latency: 27.0 ms, 95th: 48 ms, 99th: 69 ms Without prefetching (without the patch at all): throughput: 32.8 ops/sec, avg latency: 30.4 ms, 95th: 50 ms, 99th: 227 ms On average, the latency is about 10% shorter. > Port HBASE-6874 Implement prefetching for scanners from 0.89-fb > - > > Key: HBASE-8420 > URL: https://issues.apache.org/jira/browse/HBASE-8420 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, > trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch > > > This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8519) Backup master will never come up if primary master dies during initialization
[
https://issues.apache.org/jira/browse/HBASE-8519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654859#comment-13654859
]
Jean-Daniel Cryans commented on HBASE-8519:
---
What this jira describes is not the only failure mode. If you kill the master
while it's shutting down the cluster and you try to restart HBase, it will have
the same error and will go down (and the cluster won't even come up).
All you need is the master and the shutdown znodes in place when starting
HMaster.
There's a weird misconnection here:
# If you start the cluster on a clean ZK, you won't find either znodes so you
just start.
# If you start the cluster and the shutdown znode exists but not the master
znode, *you just clean it*:
{code}
// Set the cluster as up. If new RSs, they'll be waiting on this before
// going ahead with their startup.
boolean wasUp = this.clusterStatusTracker.isClusterUp();
if (!wasUp) this.clusterStatusTracker.setClusterUp();
{code}
# If you start the cluster and the master znode exists but not the shutdown,
you are a backup master.
# Finally, this jira, you have both so you assume the cluster is shutting down
and you were meant to be a backup cluster.
The tricky part here is that 2 and 4 are almost the same, except that you want
to handle the case where the whole cluster is shutting down while you're
waiting. Maybe we could check if both znodes are there before waiting and then
recognize the situation as being a start and not a stop, but I'm sure that's
going to screw someone at some point too...
Comments?
> Backup master will never come up if primary master dies during initialization
> -
>
> Key: HBASE-8519
> URL: https://issues.apache.org/jira/browse/HBASE-8519
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.7, 0.95.0
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 0.98.0
>
>
> The problem happens if primary master dies after becoming master but before
> it completes initialization and calls clusterStatusTracker.setClusterUp(),
> The backup master will try to become the master, but will shutdown itself
> promptly because it sees 'the cluster is not up'.
> This is the backup master log:
> 2013-05-09 15:08:05,568 INFO
> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
> 2013-05-09 15:08:05,573 DEBUG org.apache.hadoop.hbase.master.HMaster: HMaster
> started in backup mode. Stalling until master znode is written.
> 2013-05-09 15:08:05,589 INFO
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master
> already exists and this is not a retry
> 2013-05-09 15:08:05,590 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Adding ZNode for
> /hbase/backup-masters/xxx.com,6,1368137285373 in backup master directory
> 2013-05-09 15:08:05,595 INFO
> org.apache.hadoop.hbase.master.ActiveMasterManager: Another master is the
> active master, xxx.com,6,1368137283107; waiting to become the next active
> master
> 2013-05-09 15:09:45,006 DEBUG
> org.apache.hadoop.hbase.master.ActiveMasterManager: No master available.
> Notifying waiting threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.hbase.master.HMaster: Cluster
> went down before this master became active
> 2013-05-09 15:09:45,006 DEBUG org.apache.hadoop.hbase.master.HMaster:
> Stopping service threads
> 2013-05-09 15:09:45,006 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
> server on 6
>
> In ActiveMasterManager::blockUntilBecomingActiveMaster()
> {code}
> ..
> if (!clusterStatusTracker.isClusterUp()) {
> this.master.stop(
> "Cluster went down before this master became active");
> }
> ..
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8527) clean up code around compaction completion in HStore
Sergey Shelukhin created HBASE-8527: --- Summary: clean up code around compaction completion in HStore Key: HBASE-8527 URL: https://issues.apache.org/jira/browse/HBASE-8527 Project: HBase Issue Type: Improvement Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Priority: Minor The methods completeCompaction and it's caller are too long. Something I changed while doing something else there; putting in a separate easy-to-review JIRA to make the other future change smaller. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[ https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8154: -- Attachment: 8154-v4.txt > HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address > already in use" and retry automatically > - > > Key: HBASE-8154 > URL: https://issues.apache.org/jira/browse/HBASE-8154 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Ted Yu > Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt, 8154-v4.txt > > > see > https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ > and > https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8420: --- Status: Patch Available (was: Open) Patch v3 added configuration to control the size of max number of prefetcher threads, and max prefetched result size. > Port HBASE-6874 Implement prefetching for scanners from 0.89-fb > - > > Key: HBASE-8420 > URL: https://issues.apache.org/jira/browse/HBASE-8420 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, > trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch > > > This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8420) Port HBASE-6874 Implement prefetching for scanners from 0.89-fb
[ https://issues.apache.org/jira/browse/HBASE-8420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-8420: --- Attachment: trunk-8420_v3.patch > Port HBASE-6874 Implement prefetching for scanners from 0.89-fb > - > > Key: HBASE-8420 > URL: https://issues.apache.org/jira/browse/HBASE-8420 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Attachments: 0.94-8420_v1.patch, 0.94-8420_v2.patch, > trunk-8420_v1.patch, trunk-8420_v2.patch, trunk-8420_v3.patch > > > This should help scanner performance. We should have it in trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7380) [replication] When transferring queues, check if the peer still exists before copying the znodes
[
https://issues.apache.org/jira/browse/HBASE-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654820#comment-13654820
]
Jean-Daniel Cryans commented on HBASE-7380:
---
I posted a patch yesterday on rb: https://reviews.apache.org/r/11014/
It's much different from the one I posted for 0.94 as I saw that it was wrong.
> [replication] When transferring queues, check if the peer still exists before
> copying the znodes
>
>
> Key: HBASE-7380
> URL: https://issues.apache.org/jira/browse/HBASE-7380
> Project: HBase
> Issue Type: Bug
> Components: Replication, Usability
>Affects Versions: 0.94.3
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Fix For: 0.95.1
>
> Attachments: HBASE-7380-0.94.patch
>
>
> Right now it's a pain if you remove a peer and still have rogue queues
> because they get moved on and on and on. NodeFailoverWorker needs to run the
> check:
> bq. if (!zkHelper.getPeerClusters().containsKey(src.getPeerClusterId())) {
> before this:
> bq. SortedMap> newQueues =
> zkHelper.copyQueuesFromRS(rsZnode);
> And test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8526) Backport HBASE-5667 into 0.94
[ https://issues.apache.org/jira/browse/HBASE-8526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-8526: --- Attachment: (was: HBASE-8526-v0-0.94.patch) > Backport HBASE-5667 into 0.94 > - > > Key: HBASE-8526 > URL: https://issues.apache.org/jira/browse/HBASE-8526 > Project: HBase > Issue Type: Improvement > Components: Filters >Affects Versions: 0.94.7 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > > RegexStringComparator supports java.util.regex.Pattern flags > https://issues.apache.org/jira/browse/HBASE-5667 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8526) Backport HBASE-5667 into 0.94
[ https://issues.apache.org/jira/browse/HBASE-8526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654793#comment-13654793 ] Jean-Marc Spaggiari commented on HBASE-8526: 1) I took the wrong patch as the source 2) There is some backward wire compatibility issues... Working on that. > Backport HBASE-5667 into 0.94 > - > > Key: HBASE-8526 > URL: https://issues.apache.org/jira/browse/HBASE-8526 > Project: HBase > Issue Type: Improvement > Components: Filters >Affects Versions: 0.94.7 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > > RegexStringComparator supports java.util.regex.Pattern flags > https://issues.apache.org/jira/browse/HBASE-5667 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[
https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654783#comment-13654783
]
Hadoop QA commented on HBASE-8154:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582660/8154-v4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5624//console
This message is automatically generated.
> HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address
> already in use" and retry automatically
> -
>
> Key: HBASE-8154
> URL: https://issues.apache.org/jira/browse/HBASE-8154
> Project: HBase
> Issue Type: Test
> Components: test
>Affects Versions: 0.98.0
>Reporter: Liang Xie
>Assignee: Ted Yu
> Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt
>
>
> see
> https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/
> and
> https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7006) [MTTR] Improve Region Server Recovery Time - Distributed Log Replay
[
https://issues.apache.org/jira/browse/HBASE-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654782#comment-13654782
]
Hadoop QA commented on HBASE-7006:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582663/hbase-7006-combined-v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 31 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestHCM
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5625//console
This message is automatically generated.
> [MTTR] Improve Region Server Recovery Time - Distributed Log Replay
> ---
>
> Key: HBASE-7006
> URL: https://issues.apache.org/jira/browse/HBASE-7006
> Project: HBase
> Issue Type: New Feature
> Components: MTTR
>Reporter: stack
>Assignee: Jeffrey Zhong
>Priority: Critical
> Fix For: 0.95.1
>
> Attachments: hbase-7006-combined.patch, hbase-7006-combined-v1.patch,
> hbase-7006-combined-v4.patch, hbase-7006-combined-v5.patch,
> hbase-7006-combined-v6.patch, hbase-7006-combined-v7.patch, LogSplitting
> Comparison.pdf,
> ProposaltoimprovelogsplittingprocessregardingtoHBASE-7006-v2.pdf
>
>
> Just saw interesting issue where a cluster went down hard and 30 nodes had
> 1700 WALs to replay. Replay took almost an hour. It looks like it could run
> faster that much of the time is spent zk'ing and nn'ing.
> Putting in 0.96 so it gets a look at least. Can always punt.
> Distributed Log Replay Description:
> After a region server fails, we firstly assign a failed region to another
> region server with recovering state marked in ZooKeeper. Then a
> SplitLogWorker directly replays edits from WAL(Write-Ahead-Log)s of the
> failed region server to the region after it's re-opened in the new location.
> When a region is in recovering state, it can also accept writes but no
> reads(including Append and Increment), region split or merge.
> The feature piggybacks on existing distributed log splitting framework and
> directly replay WAL edits to another region server instead of creating
> recovered.edits files.
> The advantages over existing log splitting recovered edits implementation:
> 1) Eliminate the steps to write and read recovered.edits files. There could
> be thousands of recovered.edits files are created and written concurrently
> during a region server recovery. Many small r
[jira] [Commented] (HBASE-8525) Use sleep multilier when choosing sinks in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-8525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654750#comment-13654750
]
Jean-Daniel Cryans commented on HBASE-8525:
---
2 comments:
- Should this.sleepForRetries be reset after this?
- Before the patch we were sleeping even if finding sinks. Now we might wait
even more time if we spin a bit and then find them.
> Use sleep multilier when choosing sinks in ReplicationSource
>
>
> Key: HBASE-8525
> URL: https://issues.apache.org/jira/browse/HBASE-8525
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Priority: Trivial
> Fix For: 0.98.0, 0.94.8, 0.95.1
>
> Attachments: 8525-0.94.txt
>
>
> Currently we see this every second. Filling up the log:
> {code}
> 2013-05-10 18:36:00,766 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server ist6-mnds1-2-sfm.ops.sfdc.net/10.224.156.197:2181. Will
> not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> 2013-05-10 18:36:00,767 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 2013-05-10 18:36:01,868 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server ist6-mnds1-4-sfm.ops.sfdc.net/10.224.156.199:2181. Will
> not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> 2013-05-10 18:36:01,870 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 2013-05-10 18:36:01,971 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server ist6-mnds1-3-sfm.ops.sfdc.net/10.224.156.198:2181. Will
> not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> {code}
> Patch is trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8355) BaseRegionObserver#pre(Compact|Flush|Store)ScannerOpen returns null
[ https://issues.apache.org/jira/browse/HBASE-8355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-8355: --- Status: Open (was: Patch Available) > BaseRegionObserver#pre(Compact|Flush|Store)ScannerOpen returns null > --- > > Key: HBASE-8355 > URL: https://issues.apache.org/jira/browse/HBASE-8355 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.98.0, 0.94.8, 0.95.1 >Reporter: Andrew Purtell >Assignee: Jesse Yates > Attachments: 8355-0.94.patch, 8355.patch, hbase-8355-trunk-v0.patch, > hbase-8355-trunk-v1.patch > > > As pointed out in https://github.com/forcedotcom/phoenix/pull/131, > BaseRegionObserver#preCompactScannerOpen returns null by default, which hoses > any coprocessors down the line, making override of this method mandatory. The > fix is trivial, patch coming momentarily. > Update: > This same behavior is present in the Flush and Store versions of the same > method - this should all be moved to the proposed 'return passed scanner' > default behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8355) BaseRegionObserver#pre(Compact|Flush|Store)ScannerOpen returns null
[ https://issues.apache.org/jira/browse/HBASE-8355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-8355: --- Status: Patch Available (was: Open) > BaseRegionObserver#pre(Compact|Flush|Store)ScannerOpen returns null > --- > > Key: HBASE-8355 > URL: https://issues.apache.org/jira/browse/HBASE-8355 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.98.0, 0.94.8, 0.95.1 >Reporter: Andrew Purtell >Assignee: Jesse Yates > Attachments: 8355-0.94.patch, 8355.patch, hbase-8355-trunk-v0.patch, > hbase-8355-trunk-v1.patch > > > As pointed out in https://github.com/forcedotcom/phoenix/pull/131, > BaseRegionObserver#preCompactScannerOpen returns null by default, which hoses > any coprocessors down the line, making override of this method mandatory. The > fix is trivial, patch coming momentarily. > Update: > This same behavior is present in the Flush and Store versions of the same > method - this should all be moved to the proposed 'return passed scanner' > default behavior. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6929) Publish Hbase 0.94 artifacts build against hadoop-2.0
[ https://issues.apache.org/jira/browse/HBASE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654744#comment-13654744 ] Roshan Naik commented on HBASE-6929: checking to see if hbase is still planning to have hadoop2 binaries published for 0.96 ? > Publish Hbase 0.94 artifacts build against hadoop-2.0 > - > > Key: HBASE-6929 > URL: https://issues.apache.org/jira/browse/HBASE-6929 > Project: HBase > Issue Type: Task > Components: build >Affects Versions: 0.94.2 >Reporter: Enis Soztutar > Attachments: 6929.txt, hbase-6929_v2.patch > > > Downstream projects (flume, hive, pig, etc) depends on hbase, but since the > hbase binaries build with hadoop-2.0 are not pushed to maven, they cannot > depend on them. AFAIK, hadoop 1 and 2 are not binary compatible, so we should > also push hbase jars build with hadoop2.0 profile into maven, possibly with > version string like 0.94.2-hadoop2.0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8154) HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address already in use" and retry automatically
[ https://issues.apache.org/jira/browse/HBASE-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654741#comment-13654741 ] Jeffrey Zhong commented on HBASE-8154: -- +1 on the v4 patch. Thanks! > HBaseTestingUtility.startMiniDFSClusterForTestHLog should handle "Address > already in use" and retry automatically > - > > Key: HBASE-8154 > URL: https://issues.apache.org/jira/browse/HBASE-8154 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 0.98.0 >Reporter: Liang Xie >Assignee: Ted Yu > Attachments: 8154-v2.txt, 8154-v3.txt, 8154-v4.txt > > > see > https://builds.apache.org/job/PreCommit-HBASE-Build/4906//testReport/org.apache.hadoop.hbase.regionserver.wal/TestHLog/testAppendClose/ > and > https://issues.apache.org/jira/browse/HBASE-7845?focusedCommentId=13607468&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13607468 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8525) Use sleep multilier when choosing sinks in ReplicationSource
[
https://issues.apache.org/jira/browse/HBASE-8525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-8525:
-
Attachment: 8525-0.94.txt
Simple patch. A bit more complicated as we want to percolate an
InterruptedException up to the called in this case (so that the thread can be
ended).
> Use sleep multilier when choosing sinks in ReplicationSource
>
>
> Key: HBASE-8525
> URL: https://issues.apache.org/jira/browse/HBASE-8525
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Priority: Trivial
> Fix For: 0.98.0, 0.94.8, 0.95.1
>
> Attachments: 8525-0.94.txt
>
>
> Currently we see this every second. Filling up the log:
> {code}
> 2013-05-10 18:36:00,766 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server ist6-mnds1-2-sfm.ops.sfdc.net/10.224.156.197:2181. Will
> not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> 2013-05-10 18:36:00,767 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 2013-05-10 18:36:01,868 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server ist6-mnds1-4-sfm.ops.sfdc.net/10.224.156.199:2181. Will
> not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> 2013-05-10 18:36:01,870 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 2013-05-10 18:36:01,971 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server ist6-mnds1-3-sfm.ops.sfdc.net/10.224.156.198:2181. Will
> not attempt to authenticate using SASL (Unable to locate a login
> configuration)
> {code}
> Patch is trivial.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
