[jira] [Created] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
Aleksandr Shulman created HBASE-6904: Summary: In the HBase shell, an error is thrown that states replication-related znodes already exist Key: HBASE-6904 URL: https://issues.apache.org/jira/browse/HBASE-6904 Project: HBase Issue Type: Bug Components: Replication, Zookeeper Affects Versions: 0.92.1, 0.92.0 Reporter: Aleksandr Shulman On a replication-enabled cluster, querying the list_peers produces the error lines shown below. It doesn't appear that anything is broken in terms of functionality. Stack trace: hbase(main):001:0 list_peers 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node /hbase/replication/peers already exists and this is not a retry 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node /hbase/replication/rs already exists and this is not a retry PEER ID CLUSTER KEY 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6904) In the HBase shell, an error is thrown that states replication-related znodes already exist
[ https://issues.apache.org/jira/browse/HBASE-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang updated HBASE-6904: - Priority: Minor (was: Major) In the HBase shell, an error is thrown that states replication-related znodes already exist --- Key: HBASE-6904 URL: https://issues.apache.org/jira/browse/HBASE-6904 Project: HBase Issue Type: Bug Components: Replication, Zookeeper Affects Versions: 0.92.0, 0.92.1 Reporter: Aleksandr Shulman Priority: Minor On a replication-enabled cluster, querying the list_peers produces the error lines shown below. It doesn't appear that anything is broken in terms of functionality. Stack trace: hbase(main):001:0 list_peers 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node /hbase/replication/peers already exists and this is not a retry 12/09/29 14:41:03 ERROR zookeeper.RecoverableZooKeeper: Node /hbase/replication/rs already exists and this is not a retry PEER ID CLUSTER KEY 0 row(s) in 0.4650 seconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6846) BitComparator bug - ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466677#comment-13466677
]
Lucian George Iordache commented on HBASE-6846:
---
1. Right now, if length1, the method compareTo returns 1, so it considers that
the values are not equal. If I were to choose if this is right, I would choose
to consider the values equal when the length is 0 (2 byte arrays with length 0
are equal) and to consider them not equal when the provided length is 0
(problem when providing the length to verify). What do you think?
2. What does the message The patch does not appear to apply with p0 to p2
PATCH APPLICATION FAILED mean? I've generated the patch using svn diff
HBASE-6846.patch
BitComparator bug - ArrayIndexOutOfBoundsException
--
Key: HBASE-6846
URL: https://issues.apache.org/jira/browse/HBASE-6846
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.94.1
Environment: HBase 0.94.1 + Hadoop 2.0.0-cdh4.0.1
Reporter: Lucian George Iordache
Attachments: HBASE-6846.patch
The HBase 0.94.1 BitComparator introduced a bug in the method compareTo:
@Override
public int compareTo(byte[] value, int offset, int length) {
if (length != this.value.length) {
return 1;
}
int b = 0;
//Iterating backwards is faster because we can quit after one non-zero
byte.
for (int i = value.length - 1; i = 0 b == 0; i--) {
switch (bitOperator) {
case AND:
b = (this.value[i] value[i+offset]) 0xff;
break;
case OR:
b = (this.value[i] | value[i+offset]) 0xff;
break;
case XOR:
b = (this.value[i] ^ value[i+offset]) 0xff;
break;
}
}
return b == 0 ? 1 : 0;
}
I've encountered this problem when using a BitComparator with a configured
this.value.length=8, and in the HBase table there were KeyValues with
keyValue.getBuffer().length=207911 bytes. In this case:
for (int i = 207910; i = 0 b == 0; i--) {
switch (bitOperator) {
case AND:
b = (this.value[207910] ... == ArrayIndexOutOfBoundsException
break;
That loop should use:
for (int i = length - 1; i = 0 b == 0; i--) { (or this.value.length.)
Should I provide a patch for correcting the problem?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466682#comment-13466682
]
ramkrishna.s.vasudevan commented on HBASE-6881:
---
@Jimmy
The case here is mainly in case of one RS. And i too feel trying to same RS
should work out fine. I feel this is ok as per the current patch. Am +1 on
commit too.
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6846) BitComparator bug - ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-6846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466689#comment-13466689
]
Lucian George Iordache commented on HBASE-6846:
---
About 1: I took a better look, and the BitComparator says only if the bit
operation on values a and b is different than 0. So, the method compareTo will
return 1 if a = b = {0, 0}. So maybe for length 1 we should return 1 (this
happens right now, in the current implementation)
BitComparator bug - ArrayIndexOutOfBoundsException
--
Key: HBASE-6846
URL: https://issues.apache.org/jira/browse/HBASE-6846
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.94.1
Environment: HBase 0.94.1 + Hadoop 2.0.0-cdh4.0.1
Reporter: Lucian George Iordache
Attachments: HBASE-6846.patch
The HBase 0.94.1 BitComparator introduced a bug in the method compareTo:
@Override
public int compareTo(byte[] value, int offset, int length) {
if (length != this.value.length) {
return 1;
}
int b = 0;
//Iterating backwards is faster because we can quit after one non-zero
byte.
for (int i = value.length - 1; i = 0 b == 0; i--) {
switch (bitOperator) {
case AND:
b = (this.value[i] value[i+offset]) 0xff;
break;
case OR:
b = (this.value[i] | value[i+offset]) 0xff;
break;
case XOR:
b = (this.value[i] ^ value[i+offset]) 0xff;
break;
}
}
return b == 0 ? 1 : 0;
}
I've encountered this problem when using a BitComparator with a configured
this.value.length=8, and in the HBase table there were KeyValues with
keyValue.getBuffer().length=207911 bytes. In this case:
for (int i = 207910; i = 0 b == 0; i--) {
switch (bitOperator) {
case AND:
b = (this.value[207910] ... == ArrayIndexOutOfBoundsException
break;
That loop should use:
for (int i = length - 1; i = 0 b == 0; i--) { (or this.value.length.)
Should I provide a patch for correcting the problem?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466693#comment-13466693
]
Ted Yu commented on HBASE-6317:
---
HBASE-6381 has gone in.
Shall we consider checking in this patch ?
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466694#comment-13466694 ] Alexander Alten-Lorenz commented on HBASE-6694: --- Michael, Jon - can we mark as solved? Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds Key: HBASE-6694 URL: https://issues.apache.org/jira/browse/HBASE-6694 Project: HBase Issue Type: Task Reporter: stack Assignee: Alexander Alten-Lorenz Attachments: HBASE-6694.patch From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-6317:
--
Attachment: HBASE-6317_trunk_4.patch
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466729#comment-13466729
]
rajeshbabu commented on HBASE-6317:
---
Rebased patch after HBASE-6381 addressing Jimmy's comment. Uploaded same to RB:
https://reviews.apache.org/r/6011/
Please review and provide comments/suggestions.
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466740#comment-13466740
]
Hadoop QA commented on HBASE-6317:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547206/HBASE-6317_trunk_4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
149 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2974//console
This message is automatically generated.
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6861) HFileOutputFormat set TIMERANGE wrongly
[
https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Morozov updated HBASE-6861:
--
Attachment: diff
This is how I've fixed this issue for myself.
HFileOutputFormat set TIMERANGE wrongly
---
Key: HBASE-6861
URL: https://issues.apache.org/jira/browse/HBASE-6861
Project: HBase
Issue Type: Bug
Reporter: Eugene Morozov
Priority: Minor
Attachments: diff
In case if timestamps for KeyValues specified differently for different
column families, then TIMERANGEs of both HFiles would be wrong.
Example (in pseudo code): my reducer has a condition
if ( condition ) {
keyValue = new KeyValue(.., CF1, .., timestamp, ..);
} else {
keyValue = new KeyValue(.., CF2, .., ..); // - no timestamp
}
context.write( keyValue );
These two keyValues would be written into two different HFiles.
But the code, which is actually write do the following:
// we now have the proper HLog writer. full steam ahead
kv.updateLatestStamp(this.now);
trt.includeTimestamp(kv);
wl.writer.append(kv);
Basically, two HFiles shares the same instance of trt (TimeRangeTracker),
which leads to the same TIMERANGEs of both of them. Which is definitely
incorrect, because first HFile must have TIMERANGE=timestamp...timestamp,
cause we do not write any other timestamps there. And another HFile must have
TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6905) Regionserver Metrics reporting as negative numbers
Ron Buckley created HBASE-6905: -- Summary: Regionserver Metrics reporting as negative numbers Key: HBASE-6905 URL: https://issues.apache.org/jira/browse/HBASE-6905 Project: HBase Issue Type: Bug Affects Versions: 0.92.1 Reporter: Ron Buckley Priority: Minor Our HBase cluster has been up since May. Over the weekend one of our monitors started barking because the readRequestsCount is being reported as negative. requestsPerSecond=17, numberOfOnlineRegions=81, numberOfStores=91, numberOfStorefiles=115, storefileIndexSizeMB=2, rootIndexSizeKB=2497, totalStaticIndexSizeKB=197243, totalStaticBloomSizeKB=218013, memstoreSizeMB=286, readRequestsCount=-1820030847, writeRequestsCount=326458652, compactionQueueSize=0, flushQueueSize=0, usedHeapMB=6303, maxHeapMB=10214, blockCacheSizeMB=4137.7, blockCacheFreeMB=969.52, blockCacheCount=60445, blockCacheHitCount=3040378952, blockCacheMissCount=501619888, blockCacheEvictedCount=134409709, blockCacheHitRatio=85%, blockCacheHitCachingRatio=95%, hdfsBlocksLocalityIndex=100 These are the current counts our regiion servers are reporting. hadoopsup@wchddb01pxdu:~/hbaseTools$ ./getHBaseMetric.sh readRequestsCount | sort -k +2 -t = -n wchddb41pxdu.prod.oclc.org:9004 readRequestsCount=-2140115894 wchddb18pxdu.prod.oclc.org:9004 readRequestsCount=-2129290048 wchddb33pxdu.prod.oclc.org:9004 readRequestsCount=-1938060269 wchddb22pxdu.prod.oclc.org:9004 readRequestsCount=-1819997297 wchddb37pxdu.prod.oclc.org:9004 readRequestsCount=-1361713407 wchddb39pxdu.prod.oclc.org:9004 readRequestsCount=-1346438303 wchddb15pxdu.prod.oclc.org:9004 readRequestsCount=23724328 wchddb17pxdu.prod.oclc.org:9004 readRequestsCount=26673900 wchddb31pxdu.prod.oclc.org:9004 readRequestsCount=26753467 wchddb08pxdu.prod.oclc.org:9004 readRequestsCount=29632942 wchddb50pxdu.prod.oclc.org:9004 readRequestsCount=42054983 wchddb43pxdu.prod.oclc.org:9004 readRequestsCount=91783220 wchddb45pxdu.prod.oclc.org:9004 readRequestsCount=276019000 wchddb12pxdu.prod.oclc.org:9004 readRequestsCount=332978804 wchddb14pxdu.prod.oclc.org:9004 readRequestsCount=414414026 wchddb25pxdu.prod.oclc.org:9004 readRequestsCount=714715497 wchddb46pxdu.prod.oclc.org:9004 readRequestsCount=854608441 wchddb28pxdu.prod.oclc.org:9004 readRequestsCount=1056464575 wchddb40pxdu.prod.oclc.org:9004 readRequestsCount=1142317805 wchddb11pxdu.prod.oclc.org:9004 readRequestsCount=1192703412 wchddb07pxdu.prod.oclc.org:9004 readRequestsCount=1324556090 wchddb23pxdu.prod.oclc.org:9004 readRequestsCount=1345624237 wchddb47pxdu.prod.oclc.org:9004 readRequestsCount=1354096261 wchddb38pxdu.prod.oclc.org:9004 readRequestsCount=1494019278 wchddb27pxdu.prod.oclc.org:9004 readRequestsCount=1556667925 wchddb10pxdu.prod.oclc.org:9004 readRequestsCount=1578493607 wchddb21pxdu.prod.oclc.org:9004 readRequestsCount=1621338673 wchddb29pxdu.prod.oclc.org:9004 readRequestsCount=1623831234 wchddb48pxdu.prod.oclc.org:9004 readRequestsCount=1643042108 wchddb24pxdu.prod.oclc.org:9004 readRequestsCount=1662372209 wchddb34pxdu.prod.oclc.org:9004 readRequestsCount=1709785225 wchddb20pxdu.prod.oclc.org:9004 readRequestsCount=1710768413 wchddb42pxdu.prod.oclc.org:9004 readRequestsCount=1710792044 wchddb30pxdu.prod.oclc.org:9004 readRequestsCount=1742485411 wchddb26pxdu.prod.oclc.org:9004 readRequestsCount=1819557634 wchddb49pxdu.prod.oclc.org:9004 readRequestsCount=1856456352 wchddb44pxdu.prod.oclc.org:9004 readRequestsCount=1858474961 wchddb06pxdu.prod.oclc.org:9004 readRequestsCount=1894403007 wchddb13pxdu.prod.oclc.org:9004 readRequestsCount=1938887475 wchddb09pxdu.prod.oclc.org:9004 readRequestsCount=1970495619 wchddb05pxdu.prod.oclc.org:9004 readRequestsCount=1998165666 wchddb16pxdu.prod.oclc.org:9004 readRequestsCount=2003054336 wchddb04pxdu.prod.oclc.org:9004 readRequestsCount=2015847806 wchddb32pxdu.prod.oclc.org:9004 readRequestsCount=2134753437 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466886#comment-13466886
]
Flavio Junqueira commented on HBASE-5937:
-
Thanks a lot for the review, Stack, and Ted for the additions. Here are some
more comments:
bq. How is HLogSplitter going to work w/ different implementations? Currently
you punt and refer to FSHLog?
The splitter might not be necessary for every implementation, so I don't think
it is a good idea to make it part of HLog. Since BookKeeper is optimized for
writing multiple parallel streams efficiently, my perspective is that we want
to benefit from this feature to avoid having to split upon recovery. This is my
understanding at least, and it would be great to have a clarification if it is
incorrect.
If we assume that a splitter is necessary for every filesystem-based wal, then
we could have another interface extending HLog and defining methods related to
splitting. If not, then the splitter could specific to FSHLog. How does it
sound?
bq. Yeah, whats this crazy thing supposed to do? resetLogReaderClass
I can't remember now where I got it from, but it is used in TestHLogSplit only
to reset the class we use for the HLog reader. We have the ability to configure
the class we use for the reader and the writer.
bq. FYI, next time, you could use Bytes.toBytes in HBase under util to do this:
HBASE::CACHEFLUSH.getBytes(HConstants.UTF8_ENCODING);
Ok, thanks for the hint. I don't think I have added that statement, but I'm
happy to change it.
{quote}
The following from HLogUtil I'd think belong in an implementation:
validateHLogFilename
getReader // Why this in util and in factory too? or, its commented out
along w/ createWriter?
getHLogDirectoryName
getRegionDirRecoveredEditsDir // Can do this later... seems like I ask the
implementation for reader on the edits...
moveAsideBadEditsFile
getSplitEditFilesSorted // Yeah, this would be in the HLog splitting
implementation... or not exposed
{quote}
Are you suggesting that we make them static methods of say FSHLog instead of
having them in HLogUtil? They seem to be specific to the current trunk
implementation of HLog, so it seems right to do it. The idea behind HLogUtil
was to have all static methods in HLog originally. I can either move them now
or create a jira to revisit HLogUtil.
{quote}
So, what you want to do? If tests pass and you are fairly sure you have not
done anything to change our wal write speed – it doesn't look like you have
changed any of that, then if you want to do a part 1 commit and then open new
issue to address outstanding, improving WAL Interface, etc., I'd be +1 on that.
{quote}
I don't think we have changed anything that would affect your wal speed, at
least not consciously. The follow-up task I believe we need to work on, and
they could be jiras directly, are:
# Rename HLog interface to WAL as you suggested;
# Revisit methods of the HLog interface;
# Revisit methods of HLogUtil and HLogMetrics;
# Revisit HLogSplitter.
# Implement BKHLog.
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch,
HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch,
org.apache.hadoop.hbase.client.TestMultiParallel-output.txt
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6861) HFileOutputFormat set TIMERANGE wrongly
[
https://issues.apache.org/jira/browse/HBASE-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466897#comment-13466897
]
Ted Yu commented on HBASE-6861:
---
@Eugene:
Consider using the formatter from HBASE-5961.
HFileOutputFormat set TIMERANGE wrongly
---
Key: HBASE-6861
URL: https://issues.apache.org/jira/browse/HBASE-6861
Project: HBase
Issue Type: Bug
Reporter: Eugene Morozov
Priority: Minor
Attachments: diff
In case if timestamps for KeyValues specified differently for different
column families, then TIMERANGEs of both HFiles would be wrong.
Example (in pseudo code): my reducer has a condition
if ( condition ) {
keyValue = new KeyValue(.., CF1, .., timestamp, ..);
} else {
keyValue = new KeyValue(.., CF2, .., ..); // - no timestamp
}
context.write( keyValue );
These two keyValues would be written into two different HFiles.
But the code, which is actually write do the following:
// we now have the proper HLog writer. full steam ahead
kv.updateLatestStamp(this.now);
trt.includeTimestamp(kv);
wl.writer.append(kv);
Basically, two HFiles shares the same instance of trt (TimeRangeTracker),
which leads to the same TIMERANGEs of both of them. Which is definitely
incorrect, because first HFile must have TIMERANGE=timestamp...timestamp,
cause we do not write any other timestamps there. And another HFile must have
TIMERANGE=now...now by same meaning.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466901#comment-13466901 ] Ted Yu commented on HBASE-6707: --- Now there seems to be another dimension to consider: TestZooKeeperTableArchiveClient is failing in trunk (maybe due to switch to jdk 1.7) How would we know that the fix really works after it gets checked in ? TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466920#comment-13466920 ] Francis Liu commented on HBASE-6721: Stack, we'd like to get this into 0.96 (and if possible 0.94 as that's what we're planning on deploying). We should be have a usable patch in ~2 weeks. Does that sound acceptable? RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh reassigned HBASE-6906:
-
Assignee: Jonathan Hsieh
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy25.disableTable(Unknown Source)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:764)
... 12 more
{code}
[jira] [Created] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
Jonathan Hsieh created HBASE-6906:
-
Summary: TestHbckFsck#testQuarantine* tests are flakey due to
TableNotEnabledException
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Reporter: Jonathan Hsieh
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException: testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy25.disableTable(Unknown Source)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:764)
... 12 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6906:
--
Affects Version/s: 0.96.0
0.94.2
0.92.3
Fix Version/s: 0.96.0
0.94.3
0.92.3
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
[jira] [Updated] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6906:
--
Attachment: hbase-6906-94.patch
applies on 92/94 branches. for trunk, cd hbase-server and then apply.
all quarantine tests passed 10x on 0.92 branch.
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at
[jira] [Updated] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6906:
--
Status: Patch Available (was: Open)
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
[jira] [Updated] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6906:
--
Attachment: hbase-6906.patch
attached trunk version.
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466933#comment-13466933 ] Francis Liu commented on HBASE-5498: Thanks for the feedback guys. I'll post an updated patch sometime this week. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.3, 0.96.0 Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466938#comment-13466938 ] Francis Liu commented on HBASE-5498: @Ted, Is coprocessorService an 0.96 enhancement? I'm hoping we can address 0.96 streamlining in a separate patch since there's a bunch of work to do there already (removing some workarounds to prevent security bleeding into the core code). Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.3, 0.96.0 Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466947#comment-13466947 ] Ted Yu commented on HBASE-5498: --- bq. Is coprocessorService an 0.96 enhancement Yes. According to recent discussion on mailing list, support for the existing endpoint would be dropped in 0.96 If this patch adds another endpoint in the old way, 0.96 would be blocked further. @Stack, @Gary: What do you think ? Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.3, 0.96.0 Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5261) Update HBase for Java 7
[ https://issues.apache.org/jira/browse/HBASE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466949#comment-13466949 ] Kumar Ravi commented on HBASE-5261: --- I ran all the categories (Small, Medium and Large) of Junit tests on Hbase-0.94.1 (Revision 1392444) built using IBM Java 7 and all the categories ran without any failures. Update HBase for Java 7 --- Key: HBASE-5261 URL: https://issues.apache.org/jira/browse/HBASE-5261 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin We need to make sure that HBase compiles and works with JDK 7. Once we verify it is reasonably stable, we can explore utilizing the G1 garbage collector. When all deployments are ready to move to JDK 7, we can start using new language features, but in the transition period we will need to maintain a codebase that compiles both with JDK 6 and JDK 7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466953#comment-13466953 ] stack commented on HBASE-5937: -- bq. The splitter might not be necessary for every implementation, so I don't think it is a good idea to make it part of HLog. Its fine it doesn't need to split but Master needs some way of figuring that w/ a particular implemenation, there is no need to split. How will you do this? bq. I can't remember now where I got it from, but it is used in TestHLogSplit only to reset the class we use for the HLog reader. We have the ability to configure the class we use for the reader and the writer. If only for that test, I'd say we file an issue to fix this ugly addition to HLog only used in a single test bq. Ok, thanks for the hint. I don't think I have added that statement, but I'm happy to change it. Not important. Just an FYI. bq. Are you suggesting that we make them static methods of say FSHLog instead of having them in HLogUtil? They seem to be me like they belong in the Interface... getReader... or, if FSHLog particular, in FSHLog. bq. The idea behind HLogUtil was to have all static methods in HLog originally. To my mind, class named HLogUtil is utility that could be used by any HLog implementation. If that is not the case, call it FSHLogUtil? You want to do your list in other JIRAs and get this patch committed now (do you want to do one last update)? Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466964#comment-13466964 ] Jonathan Hsieh commented on HBASE-6694: --- [~alo.alt] Thanks for the ping and the test results -- looks good to me. I'll commit and resolve. Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds Key: HBASE-6694 URL: https://issues.apache.org/jira/browse/HBASE-6694 Project: HBase Issue Type: Task Reporter: stack Assignee: Alexander Alten-Lorenz Attachments: HBASE-6694.patch From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6694: -- Status: Patch Available (was: Open) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds Key: HBASE-6694 URL: https://issues.apache.org/jira/browse/HBASE-6694 Project: HBase Issue Type: Task Reporter: stack Assignee: Alexander Alten-Lorenz Attachments: HBASE-6694.patch From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466968#comment-13466968
]
stack commented on HBASE-6906:
--
So we just wait around till table is enabled before proceeding? If so, +1
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466972#comment-13466972 ] Jesse Yates commented on HBASE-6707: [~te...@apache.org] why would that change anything? It was already flaky, so we know the fix works when the test passes...? TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466971#comment-13466971 ] stack commented on HBASE-6721: -- That should work. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466977#comment-13466977
]
Jonathan Hsieh commented on HBASE-6906:
---
Yup, that is exactly it. Will commit.
TestHbckFsck#testQuarantine* tests are flakey due to TableNotEnabledException
-
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466981#comment-13466981 ] Ted Yu commented on HBASE-6707: --- bq. so we know the fix works when the test passes TestZooKeeperTableArchiveClient has been failing in trunk builds 3397 and 3398, but not in build #3396 where jdk 1.6 was used. TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[
https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466982#comment-13466982
]
Jonathan Hsieh commented on HBASE-6694:
---
Just tested patch and it fails. :(
{code}
Running org.apache.hadoop.hbase.mapreduce.TestImportExport
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 60.171 sec
FAILURE!
Results :
Failed tests:
testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport):
expected:3 but was:
1
Tests in error:
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport): 2
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0
{code}
Test scanner batching in export job feature HBASE-6372 AND report on
improvement HBASE-6372 adds
Key: HBASE-6694
URL: https://issues.apache.org/jira/browse/HBASE-6694
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: Alexander Alten-Lorenz
Attachments: HBASE-6694.patch
From tail of HBASE-6372, Jon had raised issue that test added did not
actually test the feature. This issue is about adding a test of HBASE-6372.
We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6906:
--
Summary: TestHBaseFsck#testQuarantine* tests are flakey due to
TableNotEnabledException (was: TestHbckFsck#testQuarantine* tests are flakey
due to TableNotEnabledException)
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at
[jira] [Updated] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-6906:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at
[jira] [Created] (HBASE-6907) KeyValue equals and compareTo methods should match
Matt Corgan created HBASE-6907: -- Summary: KeyValue equals and compareTo methods should match Key: HBASE-6907 URL: https://issues.apache.org/jira/browse/HBASE-6907 Project: HBase Issue Type: Bug Components: util Reporter: Matt Corgan Fix For: 0.96.0 KeyValue.KVComparator includes the memstoreTS when comparing, however the KeyValue.equals() method ignores the memstoreTS. The Comparator interface has always specified that comparator return 0 when equals would return true and vice versa. Obeying that rule has been sort of optional in the past, but Java 7 introduces a new default collection sorting algorithm called Tim Sort which relies on that behavior. http://bugs.sun.com/view_bug.do?bug_id=6804124 Possible problem spots: * there's a Collections.sort(KeyValues) in RedundantKVGenerator.generateTestKeyValues(..) * TestColumnSeeking compares two collections of KeyValues using the containsAll method. It is intentionally ignoring memstoreTS, so will need an alternative method for comparing the two collections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6901) Store file compactSelection throws ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HBASE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6901: --- Resolution: Fixed Fix Version/s: 0.96.0 0.94.3 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Stack for review. Added a unit test as suggested and integrated into trunk and 0.94. Store file compactSelection throws ArrayIndexOutOfBoundsException - Key: HBASE-6901 URL: https://issues.apache.org/jira/browse/HBASE-6901 Project: HBase Issue Type: Bug Components: HFile Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.94.3, 0.96.0 Attachments: trunk-6901.patch When setting hbase.mapreduce.hfileoutputformat.compaction.exclude to true, and run compaction to exclude bulk loaded files could cause ArrayIndexOutOfBoundsException since all files are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6476) Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge equivalent
[ https://issues.apache.org/jira/browse/HBASE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466991#comment-13466991 ] Chris Trezzo commented on HBASE-6476: - I'll take a look. Thanks! Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge equivalent - Key: HBASE-6476 URL: https://issues.apache.org/jira/browse/HBASE-6476 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Chris Trezzo Priority: Minor Fix For: 0.96.0 Attachments: 6476.txt, 6476-v2.txt, 6476-v2.txt, 6476v3.txt There are still some areas where System.currentTimeMillis() is used in HBase. In order to make all parts of the code base testable and (potentially) to be able to configure HBase's notion of time, this should be generally be replaced with EnvironmentEdgeManager.currentTimeMillis(). How hard would it be to add a maven task that checks for that, so we do not introduce System.currentTimeMillis back in the future? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6907) KeyValue equals and compareTo methods should match
[ https://issues.apache.org/jira/browse/HBASE-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6907: -- Priority: Critical (was: Major) KeyValue equals and compareTo methods should match -- Key: HBASE-6907 URL: https://issues.apache.org/jira/browse/HBASE-6907 Project: HBase Issue Type: Bug Components: util Reporter: Matt Corgan Priority: Critical Fix For: 0.96.0 KeyValue.KVComparator includes the memstoreTS when comparing, however the KeyValue.equals() method ignores the memstoreTS. The Comparator interface has always specified that comparator return 0 when equals would return true and vice versa. Obeying that rule has been sort of optional in the past, but Java 7 introduces a new default collection sorting algorithm called Tim Sort which relies on that behavior. http://bugs.sun.com/view_bug.do?bug_id=6804124 Possible problem spots: * there's a Collections.sort(KeyValues) in RedundantKVGenerator.generateTestKeyValues(..) * TestColumnSeeking compares two collections of KeyValues using the containsAll method. It is intentionally ignoring memstoreTS, so will need an alternative method for comparing the two collections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466996#comment-13466996 ] Jesse Yates commented on HBASE-6707: Yeah, but the test didn't always passing so we try to fix it, and then if it flaps try fixing it again. The nature of flappers is such that you can' be sure that it will always pass anyways (see the above comments on this thread), so its more of a wait and see. However, if the test is always failing and then we fix it such that its not, that's a net win. TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466993#comment-13466993 ] Gary Helmling commented on HBASE-5498: -- Re: SecureBulkLoadEndpoint and CoprocessorService Yes the plan is to remove CoprocessorProtocol support (the old Writable implementations) for 0.96 -- see HBASE-6895. If this patch is otherwise ready (haven't reviewed it myself), I'd suggest opening a new issue to deal with converting SecureBulkLoadEndpoint to a PB/CoprocessorService implementation, and link that to HBASE-6895 as a blocker for 0.96. That way the 0.94 and 0.96 versions of this patch don't diverge and the PB conversion only needs to be addressed for 0.96. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.3, 0.96.0 Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466998#comment-13466998 ] Enis Soztutar commented on HBASE-5937: -- bq. I think I said this in another jira, but the plan we proposed was to make of HLog an interface and expose all public methods first. This way it will be simpler to revisit the current public methods and redesign the interface. Making of HLog an interface and redesigning it all at once might be too messy. I was also gonna suggest to abstract away implementation specific method calls (splits, etc), but agreed that this patch may become unmanageable. Let's do it your way of just extracting the interface, then we can work on the public api of the hlog. Refactor HLog into an interface. Key: HBASE-5937 URL: https://issues.apache.org/jira/browse/HBASE-5937 Project: HBase Issue Type: Sub-task Reporter: Li Pi Assignee: Flavio Junqueira Priority: Minor Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, org.apache.hadoop.hbase.client.TestMultiParallel-output.txt What the summary says. Create HLog interface. Make current implementation use it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467001#comment-13467001 ] Ted Yu commented on HBASE-6707: --- bq. if the test is always failing and then we fix it such that its not, that's a net win. I am still trying to install jdk 1.7 on my MacBook. It would be nice to confirm that this fix makes TestZooKeeperTableArchiveClient pass on jdk 1.7 reliably, before integration. Thanks TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6881:
---
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467004#comment-13467004 ] Jesse Yates commented on HBASE-6707: bq. It would be nice to confirm that this fix makes TestZooKeeperTableArchiveClient pass on jdk 1.7 reliably, before integration. +1 Wouldn't HadoopQA ensure that? TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467008#comment-13467008 ] Ted Yu commented on HBASE-6707: --- It's open to discussion. If QA run returns with 196 test failures (the current number of failures on trunk), how do we interpret the test results ? TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467011#comment-13467011 ] Francis Liu commented on HBASE-5498: Re: SecureBulkLoadEndpoint and CoprocessorService That sounds like a viable proposal to me given that the comments to this patch so far has been minor. Thoughts? Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.94.3, 0.96.0 Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467009#comment-13467009
]
Hadoop QA commented on HBASE-6906:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547235/hbase-6906.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
149 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestHBaseFsck
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2975//console
This message is automatically generated.
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
[jira] [Commented] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467019#comment-13467019
]
Hadoop QA commented on HBASE-6881:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547012/trunk-6881_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2977//console
This message is automatically generated.
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467027#comment-13467027
]
Jonathan Hsieh commented on HBASE-6906:
---
Hm.. test failed but I'm pretty convinced that the breakage isn't due to this
change. I'm going to kick it off again, and if it fails I'll revert.
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at
[jira] [Commented] (HBASE-6611) Forcing region state offline cause double assignment
[ https://issues.apache.org/jira/browse/HBASE-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467028#comment-13467028 ] ramkrishna.s.vasudevan commented on HBASE-6611: --- Some comments on RB Jimmy. Thanks Forcing region state offline cause double assignment Key: HBASE-6611 URL: https://issues.apache.org/jira/browse/HBASE-6611 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: trunk-6611_v2.patch In assigning a region, assignment manager forces the region state offline if it is not. This could cause double assignment, for example, if the region is already assigned and in the Open state, you should not just change it's state to Offline, and assign it again. I think this could be the root cause for all double assignments IF the region state is reliable. After this loophole is closed, TestHBaseFsck should come up a different way to create some assignment inconsistencies, for example, calling region server to open a region directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HBASE-6881:
---
Resolution: Fixed
Fix Version/s: 0.96.0
Status: Resolved (was: Patch Available)
Thanks for Stack, Ram and Rajesh for reviewing it. Integrated to trunk.
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.96.0
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6901) Store file compactSelection throws ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HBASE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467032#comment-13467032 ] ramkrishna.s.vasudevan commented on HBASE-6901: --- Nice Store file compactSelection throws ArrayIndexOutOfBoundsException - Key: HBASE-6901 URL: https://issues.apache.org/jira/browse/HBASE-6901 Project: HBase Issue Type: Bug Components: HFile Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.94.3, 0.96.0 Attachments: trunk-6901.patch When setting hbase.mapreduce.hfileoutputformat.compaction.exclude to true, and run compaction to exclude bulk loaded files could cause ArrayIndexOutOfBoundsException since all files are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5498) Secure Bulk Load
[
https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467033#comment-13467033
]
Francis Liu commented on HBASE-5498:
[~andrew.purt...@gmail.com]
{quote}
InternalBulkLoadListener isn't necessary because there is no
BulkLoadListener – just call it BulkLoadListener?
{quote}
I added internal to disambiguate as a listener to only the actual moving of the
file and not a listener to the entire bulkload process which is what the
coprocessor hook does. I'm fine either way was worried it'll be misunderstood.
{quote}
The new '// TODO deal with viewFS' in HStore gives me concern. I think this
should be implemented, but don't have a strong opinion. There are other places
where this is going to be an issue I suspect.
{quote}
My assumption was that HBase wasn't federation compatible yet. If that is true
I think it's safe to push this to that future effort.
{quote}
In BaseRegionObserver we have //TODO this should end up as a coprocessor hook
– Those proposed hooks should be added as part of this change IMO. I don't like
the idea of BaseRegionObserver exporting something not part of the
RegionObserver interface. It is supposed to be a default implementation of that
interface not a superset.
{quote}
I didn't add this as a coprocessor hook as these methods are security only
methods which we don't want to bleed into the core code in 0.94. I added it as
a TODO so we can address this in 0.96 as part of streamlining things since we
don't need have the artificial security separation in that codebase?
{quote}
In SecureBulkLoadEndpoint we have //TODO make this configurable – This should
either be done or not?
{quote}
It is already configurable, I seem to have forgotten to remove the todo.
{quote}
In SecureTestUtil, should we be loading the SecureBulkLoad support
unconditionally? How about just for the relevant tests?
{quote}
Not sure what the downside would be? Since it is expected to always be enabled
in a secure deployment should it be always be available in the tests?
{quote}
And maybe SecureBulkLoadProxy could be moved out of LoadIncrementalHFiles to a
util class? Perhaps others will want to programatically import HFiles securely.
{quote}
I added the proxy to prevent the security code bleeding into the core code. I
extract this as a helper class if you think it's useful? It seemed to me that
LoadIncrementalHFiles was the entry point for users that wanted to do bulk load
as it does a lot of things that I believe users wouldn't want to re-roll again.
Secure Bulk Load
Key: HBASE-5498
URL: https://issues.apache.org/jira/browse/HBASE-5498
Project: HBase
Issue Type: Improvement
Components: security
Reporter: Francis Liu
Assignee: Francis Liu
Fix For: 0.94.3, 0.96.0
Attachments: HBASE-5498_94.patch, HBASE-5498_94.patch,
HBASE-5498_draft_94.patch, HBASE-5498_draft.patch, HBASE-5498_trunk.patch
Design doc:
https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load
Short summary:
Security as it stands does not cover the bulkLoadHFiles() feature. Users
calling this method will bypass ACLs. Also loading is made more cumbersome in
a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data
from user's directory to the hbase directory, which would require certain
write access privileges set.
Our solution is to create a coprocessor which makes use of AuthManager to
verify if a user has write access to the table. If so, launches a MR job as
the hbase user to do the importing (ie rewrite from text to hfiles). One
tricky part this job will have to do is impersonate the calling user when
reading the input files. We can do this by expecting the user to pass an hdfs
delegation token as part of the secureBulkLoad() coprocessor call and extend
an inputformat to make use of that token. The output is written to a
temporary directory accessible only by hbase and then bulkloadHFiles() is
called.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467036#comment-13467036
]
ramkrishna.s.vasudevan commented on HBASE-6317:
---
@Jimmy/@Stack
Pls review the patch. If it is ok i shall commit it before other AM related
stuff goes in. :)
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[
https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467039#comment-13467039
]
Hadoop QA commented on HBASE-6694:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12543625/HBASE-6694.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
149 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2976//console
This message is automatically generated.
Test scanner batching in export job feature HBASE-6372 AND report on
improvement HBASE-6372 adds
Key: HBASE-6694
URL: https://issues.apache.org/jira/browse/HBASE-6694
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: Alexander Alten-Lorenz
Attachments: HBASE-6694.patch
From tail of HBASE-6372, Jon had raised issue that test added did not
actually test the feature. This issue is about adding a test of HBASE-6372.
We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467043#comment-13467043 ] ramkrishna.s.vasudevan commented on HBASE-6721: --- Will the group info persisted in the META table? RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467045#comment-13467045
]
Ted Yu commented on HBASE-6317:
---
I ran TestMultiParallel locally based on the latest trunk patch and it passed.
@Rajesh:
You may want to run through Hadoop QA again.
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467050#comment-13467050
]
ramkrishna.s.vasudevan commented on HBASE-6881:
---
Sorry. One query. The RegionAlreadyInTransition change you removed right that
was introduced in HBASE-6438? Is it ok? Or are we going to leave the change and
address in HBASE-6896?
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.96.0
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467053#comment-13467053
]
ramkrishna.s.vasudevan commented on HBASE-6317:
---
@Ted
As i know about this patch very well the testcase change should not be related.
Thanks Ted.
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467059#comment-13467059 ] Francis Liu commented on HBASE-6721: Stack, great Ram, it'll be persisted in a file. Our thoughts were that the approach would be more invasive (and complex) if the information would be persisted in a table since we would have to know the group of a table before we can assign them to a region server and read it. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6476) Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge equivalent
[ https://issues.apache.org/jira/browse/HBASE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467064#comment-13467064 ] Chris Trezzo commented on HBASE-6476: - hmm... testSleepWithoutInterrupt seems to consistently pass when I run it locally. Investigating further. Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge equivalent - Key: HBASE-6476 URL: https://issues.apache.org/jira/browse/HBASE-6476 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Chris Trezzo Priority: Minor Fix For: 0.96.0 Attachments: 6476.txt, 6476-v2.txt, 6476-v2.txt, 6476v3.txt There are still some areas where System.currentTimeMillis() is used in HBase. In order to make all parts of the code base testable and (potentially) to be able to configure HBase's notion of time, this should be generally be replaced with EnvironmentEdgeManager.currentTimeMillis(). How hard would it be to add a maven task that checks for that, so we do not introduce System.currentTimeMillis back in the future? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467068#comment-13467068 ] ramkrishna.s.vasudevan commented on HBASE-6721: --- Ok, so if a region gets splitted then the daughter regions fall in the parents's group? So the group info file will be updated based on this info? Also groups cannot be updated once created right? RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467070#comment-13467070 ] Alexander Alten-Lorenz commented on HBASE-6694: --- The patch I submitted here is the missed test patch, I'll check what's going wrong here. The original patch was already submitted by Michael (st.Ack) in HBASE-6372 Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds Key: HBASE-6694 URL: https://issues.apache.org/jira/browse/HBASE-6694 Project: HBase Issue Type: Task Reporter: stack Assignee: Alexander Alten-Lorenz Attachments: HBASE-6694.patch From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467071#comment-13467071
]
Jimmy Xiang commented on HBASE-6881:
No problem. I think the removal is ok. The majority of HBASE-6438 is still
there.
For the removal, due to the locker, it won't be changed by the ZK event thread.
It can be changed by bulk assigner only before HBASE-6611 is in.
After HBASE-6611 is in, it won't be changed in the middle of this assign call.
We can get it back in HBASE-6896 if it turns out to be needed.
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.96.0
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467073#comment-13467073 ] Francis Liu commented on HBASE-6721: Group membership is based on tables, for a split no update will be necessary on the file. Groups can be updated. Each update will be updated in memory and flushed to the file for persistence. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated HBASE-5591: -- Attachment: ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.96.0 Attachments: ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467075#comment-13467075
]
Jimmy Xiang commented on HBASE-6317:
I posted some comment in the RB. I am +1 to get it in with the minor issue
fixed when you commit it.
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
[
https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467078#comment-13467078
]
Jonathan Hsieh commented on HBASE-6694:
---
Here's some stack traces --
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.802 sec
FAILURE!
testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time
elapsed: 14.012 sec FAILURE!
java.lang.AssertionError: expected:3 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testSimpleCase(TestImportExport.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
{code}
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 40.498 sec
FAILURE!
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time
elapsed: 14.056 sec ERROR!
java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes(TestImportExport.java:237)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
{code}
Test scanner batching in export job feature HBASE-6372 AND report on
improvement HBASE-6372 adds
Key: HBASE-6694
URL: https://issues.apache.org/jira/browse/HBASE-6694
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: Alexander Alten-Lorenz
Attachments: HBASE-6694.patch
From tail of HBASE-6372, Jon had raised issue that test added did not
actually test the feature. This issue is about adding a test of HBASE-6372.
We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[
https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467079#comment-13467079
]
Hadoop QA commented on HBASE-5591:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547256/ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2978//console
This message is automatically generated.
ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
-
Key: HBASE-5591
URL: https://issues.apache.org/jira/browse/HBASE-5591
Project: HBase
Issue Type: Improvement
Reporter: Scott Chen
Assignee: Scott Chen
Priority: Trivial
Fix For: 0.96.0
Attachments: ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch,
ASF.LICENSE.NOT.GRANTED--HBASE-5591.D2355.1.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467081#comment-13467081 ] Scott Chen commented on HBASE-5591: --- Yes, permission granted. I couldn't find the UI for granting the permission. Does this statement in the comment count? ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.96.0 Attachments: ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated HBASE-5591: -- Resolution: Fixed Status: Resolved (was: Patch Available) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.96.0 Attachments: ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5591) ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes()
[ https://issues.apache.org/jira/browse/HBASE-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467087#comment-13467087 ] stack commented on HBASE-5591: -- Thank you Scott. No check box any more. If you don't want to include it, you say so in a comment (Might be easier on you fellas with your auto-posting phabricator). Thanks for perm. ThiftServerRunner.HBaseHandler.toBytes() is identical to Bytes.getBytes() - Key: HBASE-5591 URL: https://issues.apache.org/jira/browse/HBASE-5591 Project: HBase Issue Type: Improvement Reporter: Scott Chen Assignee: Scott Chen Priority: Trivial Fix For: 0.96.0 Attachments: ASF.LICENSE.GRANTED--HBASE-5591.D2355.1.patch, ASF.LICENSE.NOT.GRANTED--HBASE-5591.D2355.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467100#comment-13467100 ] stack commented on HBASE-6707: -- [~jesse_yates] Mind updating patch for trunk? It seems to have rotted. I want to commit it because flakey TestZooKeeperTableArchiveClient is messing up clean unit test build on jdk7. Thanks boss. TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps Key: HBASE-6707 URL: https://issues.apache.org/jira/browse/HBASE-6707 Project: HBase Issue Type: Bug Components: test Reporter: Sameer Vaishampayan Assignee: Jesse Yates Priority: Critical Fix For: 0.96.0 Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, hbase-6707-v2.patch, hbase-6707-v3.patch https://builds.apache.org/job/HBase-TRUNK/3293/ Error Message Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] Stacktrace java.lang.AssertionError: Archived HFiles (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) should have gotten deleted, but didn't, remaining files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6901) Store file compactSelection throws ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HBASE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467107#comment-13467107 ] Hudson commented on HBASE-6901: --- Integrated in HBase-TRUNK #3399 (See [https://builds.apache.org/job/HBase-TRUNK/3399/]) HBASE-6901 Store file compactSelection throws ArrayIndexOutOfBoundsException (Revision 1392451) Result = FAILURE jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java Store file compactSelection throws ArrayIndexOutOfBoundsException - Key: HBASE-6901 URL: https://issues.apache.org/jira/browse/HBASE-6901 Project: HBase Issue Type: Bug Components: HFile Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.94.3, 0.96.0 Attachments: trunk-6901.patch When setting hbase.mapreduce.hfileoutputformat.compaction.exclude to true, and run compaction to exclude bulk loaded files could cause ArrayIndexOutOfBoundsException since all files are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467116#comment-13467116
]
Hudson commented on HBASE-6906:
---
Integrated in HBase-0.94 #500 (See
[https://builds.apache.org/job/HBase-0.94/500/])
HBASE-6906 TestHBaseFsck#testQuarantine* tests are flakey due to
TestNotEnabledException (Revision 1392456)
Result = FAILURE
jmhsieh :
Files :
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
[jira] [Commented] (HBASE-6901) Store file compactSelection throws ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HBASE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467117#comment-13467117 ] Hudson commented on HBASE-6901: --- Integrated in HBase-0.94 #500 (See [https://builds.apache.org/job/HBase-0.94/500/]) HBASE-6901 Store file compactSelection throws ArrayIndexOutOfBoundsException (Revision 1392459) Result = FAILURE jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java Store file compactSelection throws ArrayIndexOutOfBoundsException - Key: HBASE-6901 URL: https://issues.apache.org/jira/browse/HBASE-6901 Project: HBase Issue Type: Bug Components: HFile Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.94.3, 0.96.0 Attachments: trunk-6901.patch When setting hbase.mapreduce.hfileoutputformat.compaction.exclude to true, and run compaction to exclude bulk loaded files could cause ArrayIndexOutOfBoundsException since all files are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6103) Optimize the HBaseServer to deserialize the data for each ipc connection in parallel
[ https://issues.apache.org/jira/browse/HBASE-6103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467132#comment-13467132 ] stack commented on HBASE-6103: -- HBASE-6619 overwrites this patch and technique. Confirm after HBASE-6619 goes in and if so resolve as won't fix. Optimize the HBaseServer to deserialize the data for each ipc connection in parallel Key: HBASE-6103 URL: https://issues.apache.org/jira/browse/HBASE-6103 Project: HBase Issue Type: Improvement Components: Performance Reporter: Liyin Tang Assignee: Liyin Tang Priority: Critical Fix For: 0.96.0 Attachments: HBASE-6103-fb-89.patch Currently HBaseServer is running with a single listener thread, which is responsible for accepting the connection, reading the data from network channel, deserializing the data into writable objects and handover to the IPC handler threads. When there are multiple hbase clients connecting to the region server (HBaseServer) and reading/writing a large set of data, the listener and the respond thread will be performance bottleneck. So the solution is to deserialize the data for each ipc connection in parallel for HBaseServer BTW, it is also one of the reasons that the parallel scanning from multiple clients is far slower than single client case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2445) Clean up client retry policies
[ https://issues.apache.org/jira/browse/HBASE-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-2445: - Fix Version/s: (was: 0.96.0) We need this bad but not being worked on currently that I know of... moving this out of 0.96.0 Clean up client retry policies -- Key: HBASE-2445 URL: https://issues.apache.org/jira/browse/HBASE-2445 Project: HBase Issue Type: Improvement Components: Client Reporter: Todd Lipcon Priority: Critical Labels: moved_from_0_20_5 Right now almost all retry behavior is governed by a single parameter that determines the number of retries. In a few places, there are also conf for the number of millis to sleep between retries. This isn't quite flexible enough. If we can refactor some of the retry logic into a RetryPolicy class, we could introduce exponential backoff where appropriate, clean up some of the config, etc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6667) TestCatalogJanitor occasionally fails
[
https://issues.apache.org/jira/browse/HBASE-6667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-6667:
-
Priority: Major (was: Critical)
TestCatalogJanitor occasionally fails
-
Key: HBASE-6667
URL: https://issues.apache.org/jira/browse/HBASE-6667
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Jesse Yates
Fix For: 0.96.0
Attachments: java_6667-v0.txt, testCatalogJanitor-output.txt
Here is the OS:
Linux sea0 2.6.38-11-generic #48-Ubuntu SMP Fri Jul 29 19:02:55 UTC 2011
x86_64 x86_64 x86_64 GNU/Linux
{code}
testArchiveOldRegion(org.apache.hadoop.hbase.master.TestCatalogJanitor) Time
elapsed: 0.007 sec FAILURE!
java.lang.AssertionError: Not the same number of current files
Expected (2): Gotten (0):
Not Found:
_store0
_store1
Extra:
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at
org.apache.hadoop.hbase.util.HFileArchiveTestingUtil.assertArchiveEqualToOriginal(HFileArchiveTestingUtil.java:132)
at
org.apache.hadoop.hbase.util.HFileArchiveTestingUtil.assertArchiveEqualToOriginal(HFileArchiveTestingUtil.java:95)
at
org.apache.hadoop.hbase.master.TestCatalogJanitor.testArchiveOldRegion(TestCatalogJanitor.java:623)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467147#comment-13467147
]
Hudson commented on HBASE-6906:
---
Integrated in HBase-0.92 #588 (See
[https://builds.apache.org/job/HBase-0.92/588/])
HBASE-6906 TestHBaseFsck#testQuarantine* tests are flakey due to
TestNotEnabledException (Revision 1392457)
Result = FAILURE
jmhsieh :
Files :
* /hbase/branches/0.92/CHANGES.txt
*
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[
https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467153#comment-13467153
]
stack commented on HBASE-6317:
--
I posted on rb too.. +1 on commit after addressing Jimmys' comment
Master clean start up and Partially enabled tables make region assignment
inconsistent.
---
Key: HBASE-6317
URL: https://issues.apache.org/jira/browse/HBASE-6317
Project: HBase
Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: rajeshbabu
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch,
HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch
If we have a table in partially enabled state (ENABLING) then on HMaster
restart we treat it as a clean cluster start up and do a bulk assign.
Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it
leads to region assignment problems. Analysing more on this we found that we
have better way to handle these scenarios.
{code}
if (false == checkIfRegionBelongsToDisabled(regionInfo)
false == checkIfRegionsBelongsToEnabling(regionInfo)) {
synchronized (this.regions) {
regions.put(regionInfo, regionLocation);
addToServers(regionLocation, regionInfo);
}
{code}
We dont add to regions map so that enable table handler can handle it. But
as nothing is added to regions map we think it as a clean cluster start up.
Will come up with a patch tomorrow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6858) Fix the incorrect BADVERSION checking in the recoverable zookeeper
[ https://issues.apache.org/jira/browse/HBASE-6858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6858: - Priority: Critical (was: Major) Fix the incorrect BADVERSION checking in the recoverable zookeeper -- Key: HBASE-6858 URL: https://issues.apache.org/jira/browse/HBASE-6858 Project: HBase Issue Type: Bug Components: Zookeeper Reporter: Liyin Tang Assignee: Liyin Tang Priority: Critical Fix For: 0.96.0 Attachments: HBASE-6858.patch, HBASE-6858_v2.patch, HBASE-6858_v3.patch Thanks for Stack and Kaka's reporting that there is a bug in the recoverable zookeeper when handling BADVERSION exception for setData(). It shall compare the ID payload of the data in zk with its own identifier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6804) [replication] lower the amount of logging to a more human-readable level
[ https://issues.apache.org/jira/browse/HBASE-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6804: - Priority: Critical (was: Major) [replication] lower the amount of logging to a more human-readable level Key: HBASE-6804 URL: https://issues.apache.org/jira/browse/HBASE-6804 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Priority: Critical Fix For: 0.96.0 Attachments: HBASE-6804-0.94.patch We need stop logging every time replication decides to do something. It used to be extremely useful when the code base was younger but now it should be possible to bring it down while keeping it relevant. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6758) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file
[ https://issues.apache.org/jira/browse/HBASE-6758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6758: - Priority: Critical (was: Major) [replication] The replication-executor should make sure the file that it is replicating is closed before declaring success on that file --- Key: HBASE-6758 URL: https://issues.apache.org/jira/browse/HBASE-6758 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.96.0 Attachments: 6758-1-0.92.patch, 6758-2-0.92.patch, 6758-trunk-1.patch, TEST-org.apache.hadoop.hbase.replication.TestReplication.xml I have seen cases where the replication-executor would lose data to replicate since the file hasn't been closed yet. Upon closing, the new data becomes visible. Before that happens the ZK node shouldn't be deleted in ReplicationSourceManager.logPositionAndCleanOldLogs. Changes need to be made in ReplicationSource.processEndOfFile as well (currentPath related). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6640) [0.89-fb] Allow multiple regions to be opened simultaneously
[ https://issues.apache.org/jira/browse/HBASE-6640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6640: - Priority: Critical (was: Major) [0.89-fb] Allow multiple regions to be opened simultaneously Key: HBASE-6640 URL: https://issues.apache.org/jira/browse/HBASE-6640 Project: HBase Issue Type: Improvement Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Priority: Critical Fix For: 0.96.0 Attachments: 0001-HBASE-6640-0.89-fb-Allow-multiple-regions-to-be-open.patch Allow regions to be opened in parallel. This should reduce the time it takes to replay and reopen regions in case of a unclean restart. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6205: - Fix Version/s: (was: 0.96.0) Moving out of 0.96 Support an option to keep data of dropped table for some time - Key: HBASE-6205 URL: https://issues.apache.org/jira/browse/HBASE-6205 Project: HBase Issue Type: New Feature Affects Versions: 0.94.0, 0.96.0 Reporter: chunhui shen Assignee: chunhui shen Attachments: HBASE-6205.patch, HBASE-6205v2.patch, HBASE-6205v3.patch, HBASE-6205v4.patch, HBASE-6205v5.patch User may drop table accidentally because of error code or other uncertain reasons. Unfortunately, it happens in our environment because one user make a mistake between production cluster and testing cluster. So, I just give a suggestion, do we need to support an option to keep data of dropped table for some time, e.g. 1 day In the patch: We make a new dir named .trashtables in the rood dir. In the DeleteTableHandler, we move files in dropped table's dir to trash table dir instead of deleting them directly. And Create new class TrashCleaner which will clean dropped tables if it is time out with a period check. Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6032) Port HFileBlockIndex improvement from HBASE-5987
[ https://issues.apache.org/jira/browse/HBASE-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6032: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) This was committed a good while back. Port HFileBlockIndex improvement from HBASE-5987 Key: HBASE-6032 URL: https://issues.apache.org/jira/browse/HBASE-6032 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6032-ports-5987.txt, 6032-ports-5987-v2.txt Excerpt from HBASE-5987: First, we propose to lookahead for one more block index so that the HFileScanner would know the start key value of next data block. So if the target key value for the scan(reSeekTo) is smaller than that start kv of next data block, it means the target key value has a very high possibility in the current data block (if not in current data block, then the start kv of next data block should be returned. +Indexing on the start key has some defects here+) and it shall NOT query the HFileBlockIndex in this case. On the contrary, if the target key value is bigger, then it shall query the HFileBlockIndex. This improvement shall help to reduce the hotness of HFileBlockIndex and avoid some unnecessary IdLock Contention or Index Block Cache lookup. This JIRA is to port the fix to HBase trunk, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5826) Improve sync of HLog edits
[ https://issues.apache.org/jira/browse/HBASE-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5826: - Priority: Critical (was: Major) Improve sync of HLog edits -- Key: HBASE-5826 URL: https://issues.apache.org/jira/browse/HBASE-5826 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Todd Lipcon Priority: Critical Fix For: 0.96.0 Attachments: 5826.txt, 5826-v2.txt, 5826-v3.txt, 5826-v4.txt, 5826-v5.txt HBASE-5782 solved the correctness issue for the sync of HLog edits. Todd provided a patch that would achieve higher throughput. This JIRA is a continuation of Todd's work submitted there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5716) Make HBASE-4608 easier to use
[
https://issues.apache.org/jira/browse/HBASE-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-5716:
-
Fix Version/s: (was: 0.96.0)
Moving out of 0.96... No one working on it.
Make HBASE-4608 easier to use
-
Key: HBASE-5716
URL: https://issues.apache.org/jira/browse/HBASE-5716
Project: HBase
Issue Type: Improvement
Reporter: Jean-Daniel Cryans
Assignee: Li Pi
HBASE-4608 is a nice feature but after playing with it for a while I think
the following should be fixed to make it easier to use by someone who's not a
dev:
- Add some signal that says that the feature is turned on. Right now you can
{{jstack | grep KeyValueCompression}} a couple of times and if you get a hit
you definitely know it's on, but otherwise the random user wouldn't know
without going through the jira.
- Add documentation in the reference guide. At the minimum add
{{hbase.regionserver.wal.enablecompression}} in there with a small
description. Better would be to add a section in {{Appendix B}} or something
like that and describe the functionality a bit and who it's useful for. For
example, flush from your brain the knowledge of the patch and read the name
of the configuration... now let's say you have a use case that involves
writing easily compressible values. Any normal user would believe that this
is a good tuning parameter for them, but it's just going to waste CPU cycles.
- Add some metrics like we have for HFiles where you get a clue about the
compression ratio.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5525) Truncate and preserve region boundaries option
[ https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5525: - Tags: noob Fix Version/s: (was: 0.96.0) Out of 0.96. Truncate and preserve region boundaries option -- Key: HBASE-5525 URL: https://issues.apache.org/jira/browse/HBASE-5525 Project: HBase Issue Type: New Feature Reporter: Jean-Daniel Cryans Labels: newbie, noob A tool that would be useful for testing (and maybe in prod too) would be a truncate option to keep the current region boundaries. Right now what you have to do is completely kill the table and recreate it with the correct regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5525) Truncate and preserve region boundaries option
[ https://issues.apache.org/jira/browse/HBASE-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5525: - Labels: newbie noob (was: newbie) Truncate and preserve region boundaries option -- Key: HBASE-5525 URL: https://issues.apache.org/jira/browse/HBASE-5525 Project: HBase Issue Type: New Feature Reporter: Jean-Daniel Cryans Labels: newbie, noob A tool that would be useful for testing (and maybe in prod too) would be a truncate option to keep the current region boundaries. Right now what you have to do is completely kill the table and recreate it with the correct regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5349: - Fix Version/s: (was: 0.96.0) Moving out of 0.96.0 Automagically tweak global memstore and block cache sizes based on workload --- Key: HBASE-5349 URL: https://issues.apache.org/jira/browse/HBASE-5349 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0 Reporter: Jean-Daniel Cryans Hypertable does a neat thing where it changes the size given to the CellCache (our MemStores) and Block Cache based on the workload. If you need an image, scroll down at the bottom of this link: http://www.hypertable.com/documentation/architecture/ That'd be one less thing to configure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4575) Inconsistent naming for ZK config parameters
[ https://issues.apache.org/jira/browse/HBASE-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4575: - Priority: Critical (was: Major) Inconsistent naming for ZK config parameters Key: HBASE-4575 URL: https://issues.apache.org/jira/browse/HBASE-4575 Project: HBase Issue Type: Bug Components: test, Zookeeper Affects Versions: 0.92.0, 0.94.0 Reporter: Jonathan Gray Assignee: Jonathan Gray Priority: Critical Fix For: 0.96.0 I've found some misnaming of certain ZK config options. Make them consistent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6881) All regionservers are marked offline even there is still one up
[
https://issues.apache.org/jira/browse/HBASE-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467198#comment-13467198
]
Hudson commented on HBASE-6881:
---
Integrated in HBase-TRUNK #3400 (See
[https://builds.apache.org/job/HBase-TRUNK/3400/])
HBASE-6881 All regionservers are marked offline even there is still one up
(Revision 1392467)
Result = FAILURE
jxiang :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
All regionservers are marked offline even there is still one up
---
Key: HBASE-6881
URL: https://issues.apache.org/jira/browse/HBASE-6881
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Fix For: 0.96.0
Attachments: trunk-6881.patch, trunk-6881_v3.patch
{noformat}
+RegionPlan newPlan = plan;
+if (!regionAlreadyInTransitionException) {
+ // Force a new plan and reassign. Will return null if no servers.
+ newPlan = getRegionPlan(state, plan.getDestination(), true);
+}
+if (newPlan == null) {
this.timeoutMonitor.setAllRegionServersOffline(true);
LOG.warn(Unable to find a viable location to assign region +
state.getRegion().getRegionNameAsString());
{noformat}
Here, when newPlan is null, plan.getDestination() could be up actually.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6906) TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
[
https://issues.apache.org/jira/browse/HBASE-6906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467199#comment-13467199
]
Hudson commented on HBASE-6906:
---
Integrated in HBase-TRUNK #3400 (See
[https://builds.apache.org/job/HBase-TRUNK/3400/])
HBASE-6906 TestHBaseFsck#testQuarantine* tests are flakey due to
TestNotEnabledException (Revision 1392455)
Result = FAILURE
jmhsieh :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java
TestHBaseFsck#testQuarantine* tests are flakey due to TableNotEnabledException
--
Key: HBASE-6906
URL: https://issues.apache.org/jira/browse/HBASE-6906
Project: HBase
Issue Type: Bug
Components: hbck, test
Affects Versions: 0.92.3, 0.94.2, 0.96.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: hbase-6906-94.patch, hbase-6906.patch
This test fails periodically (1 out of 10) times on our internal jenkins
instance.
{code}
FAILED TESTS
1 tests failed.
REGRESSION:
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir
Error Message:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170) at
sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597) at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
Stack Trace:
org.apache.hadoop.hbase.TableNotEnabledException:
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1345)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at
org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:766)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.deleteTable(TestHBaseFsck.java:344)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.doQuarantineTest(TestHBaseFsck.java:1351)
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testQuarantineMissingRegionDir(TestHBaseFsck.java:1433)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotEnabledException):
org.apache.hadoop.hbase.TableNotEnabledException:
testQuarantineMissingRegionDir
at
org.apache.hadoop.hbase.master.handler.DisableTableHandler.init(DisableTableHandler.java:75)
at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:1170)
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
[jira] [Created] (HBASE-6908) Pluggable Call BlockingQueue for HBaseServer
James Taylor created HBASE-6908: --- Summary: Pluggable Call BlockingQueue for HBaseServer Key: HBASE-6908 URL: https://issues.apache.org/jira/browse/HBASE-6908 Project: HBase Issue Type: New Feature Components: IPC/RPC Reporter: James Taylor Allow the BlockingQueue implementation class to be specified in the HBase config to enable different behavior than a FIFO queue. The use case we have is around fairness and starvation for big scans that are parallelized on the client. It's easy to fill up the HBase server Call BlockingQueue when processing a single parallelized scan, leadng other scans to time out. Instead, doing round robin processesing on a dequeue through a different BlockingQueue implementation will prevent this from occurring. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6299) RS starts region open while fails ack to HMaster.sendRegionOpen() causes inconsistency in HMaster's region state and a series of successive problems.
[
https://issues.apache.org/jira/browse/HBASE-6299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-6299:
-
Attachment: 6299v4.txt
Here is what I committed after running all unit tests locally.
RS starts region open while fails ack to HMaster.sendRegionOpen() causes
inconsistency in HMaster's region state and a series of successive problems.
-
Key: HBASE-6299
URL: https://issues.apache.org/jira/browse/HBASE-6299
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.6, 0.94.0
Reporter: Maryann Xue
Assignee: Maryann Xue
Priority: Critical
Fix For: 0.92.3, 0.94.3, 0.96.0
Attachments: 6299v4.txt, 6299v4.txt, 6299v4.txt, HBASE-6299.patch,
HBASE-6299-v2.patch, HBASE-6299-v3.patch
1. HMaster tries to assign a region to an RS.
2. HMaster creates a RegionState for this region and puts it into
regionsInTransition.
3. In the first assign attempt, HMaster calls RS.openRegion(). The RS
receives the open region request and starts to proceed, with success
eventually. However, due to network problems, HMaster fails to receive the
response for the openRegion() call, and the call times out.
4. HMaster attemps to assign for a second time, choosing another RS.
5. But since the HMaster's OpenedRegionHandler has been triggered by the
region open of the previous RS, and the RegionState has already been removed
from regionsInTransition, HMaster finds invalid and ignores the unassigned ZK
node RS_ZK_REGION_OPENING updated by the second attempt.
6. The unassigned ZK node stays and a later unassign fails coz
RS_ZK_REGION_CLOSING cannot be created.
{code}
2012-06-29 07:03:38,870 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for
region
CDR_STATS_TRAFFIC,13184390567|20120508|17||2|3|913,1337256975556.b713fd655fa02395496c5a6e39ddf568.;
plan=hri=CDR_STATS_TRAFFIC,13184390567|20120508|17||2|3|913,1337256975556.b713fd655fa02395496c5a6e39ddf568.,
src=swbss-hadoop-004,60020,1340890123243,
dest=swbss-hadoop-006,60020,1340890678078
2012-06-29 07:03:38,870 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
CDR_STATS_TRAFFIC,13184390567|20120508|17||2|3|913,1337256975556.b713fd655fa02395496c5a6e39ddf568.
to swbss-hadoop-006,60020,1340890678078
2012-06-29 07:03:38,870 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=M_ZK_REGION_OFFLINE, server=swbss-hadoop-002:6,
region=b713fd655fa02395496c5a6e39ddf568
2012-06-29 07:06:28,882 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=swbss-hadoop-006,60020,1340890678078,
region=b713fd655fa02395496c5a6e39ddf568
2012-06-29 07:06:32,291 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENING, server=swbss-hadoop-006,60020,1340890678078,
region=b713fd655fa02395496c5a6e39ddf568
2012-06-29 07:06:32,299 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_OPENED, server=swbss-hadoop-006,60020,1340890678078,
region=b713fd655fa02395496c5a6e39ddf568
2012-06-29 07:06:32,299 DEBUG
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED
event for
CDR_STATS_TRAFFIC,13184390567|20120508|17||2|3|913,1337256975556.b713fd655fa02395496c5a6e39ddf568.
from serverName=swbss-hadoop-006,60020,1340890678078, load=(requests=518945,
regions=575, usedHeap=15282, maxHeap=31301); deleting unassigned node
2012-06-29 07:06:32,299 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x2377fee2ae80007 Deleting existing unassigned node for
b713fd655fa02395496c5a6e39ddf568 that is in expected state RS_ZK_REGION_OPENED
2012-06-29 07:06:32,301 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
master:6-0x2377fee2ae80007 Successfully deleted unassigned node for
region b713fd655fa02395496c5a6e39ddf568 in expected state RS_ZK_REGION_OPENED
2012-06-29 07:06:32,301 DEBUG
org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: The master has
opened the region
CDR_STATS_TRAFFIC,13184390567|20120508|17||2|3|913,1337256975556.b713fd655fa02395496c5a6e39ddf568.
that was online on serverName=swbss-hadoop-006,60020,1340890678078,
load=(requests=518945, regions=575, usedHeap=15282, maxHeap=31301)
2012-06-29 07:07:41,140 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
CDR_STATS_TRAFFIC,13184390567|20120508|17||2|3|913,1337256975556.b713fd655fa02395496c5a6e39ddf568.
to serverName=swbss-hadoop-006,60020,1340890678078, load=(requests=0,
regions=575, usedHeap=0, maxHeap=0),
