[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[
https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222801#comment-14222801
]
Ashish Singhi commented on HBASE-11639:
---
{quote}
Actually the checkstyle does not account for the imports that are accounted
thro @Link in javadoc. So when we remove that as unused imports javadoc starts
complaining.
{quote}
We can remove the import and refer the class with fully qualified name in
javadoc. By this we can avoid complains from both checkstyle and javadoc.
[Visibility controller] Replicate the visibility of Cells as strings
Key: HBASE-11639
URL: https://issues.apache.org/jira/browse/HBASE-11639
Project: HBase
Issue Type: Improvement
Components: Replication, security
Affects Versions: 0.98.4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Labels: VisibilityLabels
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments: HBASE-11639_v11.patch, HBASE-11639_v13.patch,
HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch,
HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch,
HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch,
HBASE-11639_v9.patch
This issue is aimed at persisting the visibility labels as strings in the WAL
rather than Label ordinals. This would help in replicating the label
ordinals to the replication cluster as strings directly and also that after
HBASE-11553 would help because the replication cluster could have an
implementation as string based visibility labels.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[ https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222806#comment-14222806 ] ramkrishna.s.vasudevan commented on HBASE-11639: [~ashish singhi] Thanks for the input. Let me change accordingly. [Visibility controller] Replicate the visibility of Cells as strings Key: HBASE-11639 URL: https://issues.apache.org/jira/browse/HBASE-11639 Project: HBase Issue Type: Improvement Components: Replication, security Affects Versions: 0.98.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Labels: VisibilityLabels Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-11639_v11.patch, HBASE-11639_v13.patch, HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch, HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch, HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch, HBASE-11639_v9.patch This issue is aimed at persisting the visibility labels as strings in the WAL rather than Label ordinals. This would help in replicating the label ordinals to the replication cluster as strings directly and also that after HBASE-11553 would help because the replication cluster could have an implementation as string based visibility labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11639) [Visibility controller] Replicate the visibility of Cells as strings
[
https://issues.apache.org/jira/browse/HBASE-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222829#comment-14222829
]
Hadoop QA commented on HBASE-11639:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683285/HBASE-11639_v15.patch
against master branch at commit 5911c030a509b8fc6ad5a3735f2e1279712485f7.
ATTACHMENT ID: 12683285
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3808 checkstyle errors (more than the master's current 3805 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11812//console
This message is automatically generated.
[Visibility controller] Replicate the visibility of Cells as strings
Key: HBASE-11639
URL: https://issues.apache.org/jira/browse/HBASE-11639
Project: HBase
Issue Type: Improvement
Components: Replication, security
Affects Versions: 0.98.4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Labels: VisibilityLabels
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments: HBASE-11639_v11.patch, HBASE-11639_v13.patch,
HBASE-11639_v13.patch, HBASE-11639_v14.patch, HBASE-11639_v15.patch,
HBASE-11639_v2.patch, HBASE-11639_v2.patch, HBASE-11639_v3.patch,
HBASE-11639_v3.patch, HBASE-11639_v5.patch, HBASE-11639_v6.patch,
HBASE-11639_v9.patch
This issue is aimed at persisting the visibility labels as strings in the WAL
rather than Label ordinals. This would help in replicating the label
ordinals to the replication cluster as strings directly and also that after
HBASE-11553 would help because the replication cluster could have an
implementation as string based visibility labels.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12534) Wrong region location cache in client after regions are moved
[
https://issues.apache.org/jira/browse/HBASE-12534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222873#comment-14222873
]
Nicolas Liochon commented on HBASE-12534:
-
MIN_RPC_TIMEOUT is linked to operation timeout: w/o it we could send a request
w/o giving enough time to the server. As well until recently the rcp timeout
was not multithreaded safe: it was set for all calls. So may be this min
timeout saves in in the .94 .96 versions (not sure about .98). May be this
min timeout should be configurable (cf. hbase.rpc.timeout=1000, which is lower
than the mintimeout)
Wrong region location cache in client after regions are moved
-
Key: HBASE-12534
URL: https://issues.apache.org/jira/browse/HBASE-12534
Project: HBase
Issue Type: Bug
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Critical
Labels: client
Attachments: HBASE-12534-0.94-v1.diff, HBASE-12534-v1.diff
In our 0.94 hbase cluster, we found that client got wrong region location
cache and did not update it after a region is moved to another regionserver.
The reason is wrong client config and bug in RpcRetryingCaller of hbase
client.
The rpc configs are following:
{code}
hbase.rpc.timeout=1000
hbase.client.pause=200
hbase.client.operation.timeout=1200
{code}
But the client retry number is 3
{code}
hbase.client.retries.number=3
{code}
Assumed that a region is at regionserver A before, and then it is moved to
regionserver B. The client try to make a call to regionserver A and get an
NotServingRegionException. For the rety number is not 1, the region server
location cache is not cleaned. See: RpcRetryingCaller.java#141 and
RegionServerCallable.java#127
{code}
@Override
public void throwable(Throwable t, boolean retrying) {
if (t instanceof SocketTimeoutException ||
} else if (t instanceof NotServingRegionException !retrying) {
// Purge cache entries for this specific region from hbase:meta cache
// since we don't call connect(true) when number of retries is 1.
getConnection().deleteCachedRegionLocation(location);
}
}
{code}
But the call did not retry and throw an SocketTimeoutException for the time
the call will take is larger than the operation timeout.See
RpcRetryingCaller.java#152
{code}
expectedSleep = callable.sleep(pause, tries + 1);
// If, after the planned sleep, there won't be enough time left, we
stop now.
long duration = singleCallDuration(expectedSleep);
if (duration callTimeout) {
String msg = callTimeout= + callTimeout + , callDuration= +
duration +
: + callable.getExceptionMessageAdditionalDetail();
throw (SocketTimeoutException)(new
SocketTimeoutException(msg).initCause(t));
}
{code}
At last, the wrong region location will never be not cleaned up .
[~lhofhansl]
In hbase 0.94, the MIN_RPC_TIMEOUT in singleCallDuration is 2000 in default,
which trigger this bug.
{code}
private long singleCallDuration(final long expectedSleep) {
return (EnvironmentEdgeManager.currentTimeMillis() - this.globalStartTime)
+ MIN_RPC_TIMEOUT + expectedSleep;
}
{code}
But there is risk in master code too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12474) Incorrect handling of default namespace in user_permission command.
[ https://issues.apache.org/jira/browse/HBASE-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12474: Attachment: HBASE-12474-v3.patch the test was flaky because dependent on the execution order testAccessControllerRegexHandling() and testGetNamespacePermission() are creating testNamespace... but the new testAccessControllerRegexHandling() was not removing it at the end of the test. (v3 removes the created namespace tables) Incorrect handling of default namespace in user_permission command. --- Key: HBASE-12474 URL: https://issues.apache.org/jira/browse/HBASE-12474 Project: HBase Issue Type: Bug Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-12474-v3.patch, HBASE-12474.patch, HBASE-12474_v2.patch The command user_permission returns 0 rows if we specify default prefix. It works fine if we drop the prefix. This is because pattern matching of listTables doesn't take into account that nameAsString method doesn't include 'default' in it's output for the relevant tables. This method listTables is also used by delete, disable and enable commands when invoked with regular expression. Credits to [~mbertozzi] for coming up with this corner case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12564) consolidate the getTableDescriptors() semantic
Matteo Bertozzi created HBASE-12564: --- Summary: consolidate the getTableDescriptors() semantic Key: HBASE-12564 URL: https://issues.apache.org/jira/browse/HBASE-12564 Project: HBase Issue Type: Bug Components: Client, master Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0 Master getTableDescriptors() which is called by Admin.listTables() has a couple of different behaviors depending on how it is called. after HBASE-12073 with the AccessController enabled, we now get a global admin required if listTables() is called without a regex otherwise we return only the table that the user can see (we show only the tables that the user have access to, which means or the user is a global admin or it has a table-level create/admin). We probably should have the second behavior even without regex, since I should able to see my own tables. getTableDescriptors() is returning only non system tables. Tools like user_permission that are doing for each listTable(): userPerm(table) are losing the system tables, so stuff like user_permission 'hbase:acls' will not result any result. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12474) Incorrect handling of default namespace in user_permission command.
[
https://issues.apache.org/jira/browse/HBASE-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222949#comment-14222949
]
Hadoop QA commented on HBASE-12474:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683309/HBASE-12474-v3.patch
against master branch at commit 5911c030a509b8fc6ad5a3735f2e1279712485f7.
ATTACHMENT ID: 12683309
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11813//console
This message is automatically generated.
Incorrect handling of default namespace in user_permission command.
---
Key: HBASE-12474
URL: https://issues.apache.org/jira/browse/HBASE-12474
Project: HBase
Issue Type: Bug
Reporter: Srikanth Srungarapu
Assignee: Srikanth Srungarapu
Priority: Minor
Attachments: HBASE-12474-v3.patch, HBASE-12474.patch,
HBASE-12474_v2.patch
The command user_permission returns 0 rows if we specify default prefix. It
works fine if we drop the prefix. This is because pattern matching of
listTables doesn't take into account that nameAsString method doesn't include
'default' in it's output for the relevant tables. This method listTables is
also used by delete, disable and enable commands when invoked with regular
expression. Credits to [~mbertozzi] for coming up with this corner case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12563) After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility
[ https://issues.apache.org/jira/browse/HBASE-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Talat UYARER updated HBASE-12563: - Attachment: HBASE-12563-v2.patch Hi [~stack] Actually we have some test case for master server port in TestInfoServers. I updated those in TestInfoServers code. Is it Ok ? After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility - Key: HBASE-12563 URL: https://issues.apache.org/jira/browse/HBASE-12563 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.1 Reporter: Talat UYARER Assignee: Talat UYARER Attachments: HBASE-12563-v2.patch, HBASE-12563.patch When I use startMiniHBaseCluster method. It starts up a Mini Cluster for the tests. While starting It changes some configuration. For example Master's port or Region Server's. After Cluster starting We should set its new configuration to HbaseTestingUtils conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223122#comment-14223122 ] Elliott Clark commented on HBASE-12550: --- Alright checking this in unless anyone has an objection. Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-12550: -- Resolution: Fixed Fix Version/s: 0.99.2 0.98.9 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12495) Use interfaces in the shell scripts
[ https://issues.apache.org/jira/browse/HBASE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12495: --- Attachment: HBASE-12495B.patch TestShell should now pass. However, I had to revert one of two places in order to make the tests pass. admin.rb has a change from HBaseAdmin.new() to ConnectionFactory.createConnection().getAdmin() and table.rb has a change from HTable.new() to ConnectionFactory.createConnection().getTable(). That combination causes some initialization error to happen. Removing either change fixes the problem. I commented out the table.rb changes so that tests will pass. The underlying problem in initalization ought to be fixed. This isn't the first time I've seen this type of issue. Once this patch makes it into master, I'll create another issue to describe the initialization problem. Use interfaces in the shell scripts --- Key: HBASE-12495 URL: https://issues.apache.org/jira/browse/HBASE-12495 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12495.patch, HBASE-12495B.patch Change some explicit calls to HBaseAdmin and HTable to interface counterparts in the hbase shell scripts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223242#comment-14223242 ] Hudson commented on HBASE-12550: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #664 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/664/]) HBASE-12550 Check all storefiles are referenced before splitting (elliott: rev 860ddd2c22371cf1d5d75f3d94cfad350eafb34f) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12471) Task 4. replace internal ConnectionManager#{delete,get}Connection use with #close, #createConnection (0.98, 0.99) under src/main/java
[
https://issues.apache.org/jira/browse/HBASE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12471:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Three green runs on v5. Committed on the back of open +1s from @enis and
[~ndimiduk]
What I committed was v5 with this comment up top:
{code}
HBASE-12471 Task 4. replace internal
ConnectionManager#{delete,get}Connection use with #close, #createConnection
(0.98, 0.99) under src/main/java
Move from HConnection to ClusterConnection or Connection
Use unmanaged connections where we use managed previous
(used the jdk7
https://docs.oracle.com/javase/7/docs/technotes/guides/language/try-with-resources.html
idiom).
In ZKConfig, synchronize on Configuration rather than make a copy.
Making a copy we were dropping hbase configs in certain test context
(could not find the zk ensemble because default port).
In tests, some move to the new style connection setup but mostly
fixes for premature connection close or adding cleanup where it
was lacking.
{code}
There is more to do in here so can resolve the parent issue.
Task 4. replace internal ConnectionManager#{delete,get}Connection use with
#close, #createConnection (0.98, 0.99) under src/main/java
-
Key: HBASE-12471
URL: https://issues.apache.org/jira/browse/HBASE-12471
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Fix For: 2.0.0, 0.99.2
Attachments:
0001-HBASE-12471-Task-4.-replace-internal-ConnectionManag.patch, 12404v9.txt,
124471v4.txt, 12471.reapply.reapply.v5.txt, 12471.reapply.reapply.v5.txt,
12471.reapply.reapply.v5.txt, 12471.reapply.reapply.v5.txt,
12471.reapply.reapply.v5.txt, 12471.reapply.txt, 12471.reapply.v4.txt,
12471.reapplyv2.txt, 12471.reapplyv2.txt, 12471.reapplyv3.txt, 12471v10.txt,
12471v10.txt, 12471v11.txt, 12471v2.txt, 12471v3.txt, 12471v6.txt,
12471v7.txt, 12471v9.txt
Let me do this. A bunch of this was done in HBASE-12404 Let me see if can
find more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-12550: -- Attachment: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch Addendum to fix up 0.98 on hadoop 1.1 Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223272#comment-14223272 ] Elliott Clark commented on HBASE-12479: --- Seems like this might have caused a regression on tests in 0.98 branch. Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12128: --- Affects Version/s: 2.0.0 Status: Patch Available (was: In Progress) Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12563) After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility
[ https://issues.apache.org/jira/browse/HBASE-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223295#comment-14223295 ] stack commented on HBASE-12563: --- Why remove this configuration? 51 UTIL.getConfiguration().setInt(HConstants.MASTER_INFO_PORT, 0); 52 UTIL.getConfiguration().setInt(HConstants.REGIONSERVER_INFO_PORT, 0); Now the ports are not ephemeral anymore? And in the test you assert nothing? Shouldn't you at least assert you can get to the info port ui? Thanks After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility - Key: HBASE-12563 URL: https://issues.apache.org/jira/browse/HBASE-12563 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.1 Reporter: Talat UYARER Assignee: Talat UYARER Attachments: HBASE-12563-v2.patch, HBASE-12563.patch When I use startMiniHBaseCluster method. It starts up a Mini Cluster for the tests. While starting It changes some configuration. For example Master's port or Region Server's. After Cluster starting We should set its new configuration to HbaseTestingUtils conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223319#comment-14223319 ] Andrew Purtell commented on HBASE-12479: And reopen... Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223317#comment-14223317 ] Andrew Purtell commented on HBASE-12479: Yes. Someone please revert. I'm not near a computer right now. Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223345#comment-14223345 ] Hudson commented on HBASE-12550: SUCCESS: Integrated in HBase-TRUNK #5812 (See [https://builds.apache.org/job/HBase-TRUNK/5812/]) HBASE-12550 Check all storefiles are referenced before splitting (elliott: rev 0df5ed2ca6ce3758b4745f63400fd81d17107038) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12471) Task 4. replace internal ConnectionManager#{delete,get}Connection use with #close, #createConnection (0.98, 0.99) under src/main/java
[
https://issues.apache.org/jira/browse/HBASE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223343#comment-14223343
]
Hudson commented on HBASE-12471:
SUCCESS: Integrated in HBase-TRUNK #5812 (See
[https://builds.apache.org/job/HBase-TRUNK/5812/])
HBASE-12471 Task 4. replace internal ConnectionManager#{delete,get}Connection
use with #close, #createConnection (0.98, 0.99) under src/main/java (stack: rev
336c22d581fa9e892c47a6a5158ee4ee5fc1c0f8)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapper.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerWithBulkload.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController2.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* hbase-it/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationKillRS.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionLocator.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHBaseAdminNoCluster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseCluster.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
Task 4. replace internal ConnectionManager#{delete,get}Connection use with
#close, #createConnection (0.98, 0.99) under src/main/java
-
Key: HBASE-12471
URL: https://issues.apache.org/jira/browse/HBASE-12471
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Fix For: 2.0.0, 0.99.2
Attachments:
0001-HBASE-12471-Task-4.-replace-internal-ConnectionManag.patch, 12404v9.txt,
124471v4.txt, 12471.reapply.reapply.v5.txt, 12471.reapply.reapply.v5.txt,
12471.reapply.reapply.v5.txt, 12471.reapply.reapply.v5.txt,
12471.reapply.reapply.v5.txt, 12471.reapply.txt, 12471.reapply.v4.txt,
12471.reapplyv2.txt, 12471.reapplyv2.txt, 12471.reapplyv3.txt, 12471v10.txt,
12471v10.txt, 12471v11.txt, 12471v2.txt, 12471v3.txt, 12471v6.txt,
12471v7.txt, 12471v9.txt
Let me do this. A bunch of this was done in HBASE-12404 Let me see if can
find more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12563) After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility
[ https://issues.apache.org/jira/browse/HBASE-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223358#comment-14223358 ] Talat UYARER commented on HBASE-12563: -- Those are unnecessary. They are set when you are starting MiniHBaseCluster. You can look at [LocalHBaseCluster|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java#L145] I updated configuration object. And I reach the info ui on defined port with the tests. If I can reach to the info ui on updated configuration port, Is not it enough ? After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility - Key: HBASE-12563 URL: https://issues.apache.org/jira/browse/HBASE-12563 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.1 Reporter: Talat UYARER Assignee: Talat UYARER Attachments: HBASE-12563-v2.patch, HBASE-12563.patch When I use startMiniHBaseCluster method. It starts up a Mini Cluster for the tests. While starting It changes some configuration. For example Master's port or Region Server's. After Cluster starting We should set its new configuration to HbaseTestingUtils conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[
https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223361#comment-14223361
]
Hadoop QA commented on HBASE-12128:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683210/HBASE-12128.v1-2.0.patch
against master branch at commit 0df5ed2ca6ce3758b4745f63400fd81d17107038.
ATTACHMENT ID: 12683210
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3807 checkstyle errors (more than the master's current 3806 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestInterfaceAudienceAnnotations
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11815//console
This message is automatically generated.
Cache configuration and RpcController selection for Table in Connection
---
Key: HBASE-12128
URL: https://issues.apache.org/jira/browse/HBASE-12128
Project: HBase
Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Andrew Purtell
Assignee: Stephen Yuan Jiang
Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2
Attachments: HBASE-12128.v1-2.0.patch
Original Estimate: 120h
Time Spent: 72h
Remaining Estimate: 48h
Creating Table instances should be lightweight. Apps that manage their own
Connections are expected to create Tables on demand for each interaction.
However we look up values from Hadoop Configuration when constructing Table
objects for storing to some of its fields. Configuration is a heavyweight
registry that does a lot of string operations and regex matching. Method
calls into Configuration account for 48.25% of CPU time when creating the
HTable object in 0.98. Another ~48% of CPU is
[jira] [Commented] (HBASE-12563) After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility
[ https://issues.apache.org/jira/browse/HBASE-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223382#comment-14223382 ] stack commented on HBASE-12563: --- You added no asserts to the tests? They worked before you made your change, is that right? If so, is there a test you can add that fails before your code change and passes after. Fair enough on removing the explicit setting of ports to 0. After Starting up mini hbase cluster, Real Configuration of Cluster never set to HBaseTestingUtility - Key: HBASE-12563 URL: https://issues.apache.org/jira/browse/HBASE-12563 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, 0.99.1 Reporter: Talat UYARER Assignee: Talat UYARER Attachments: HBASE-12563-v2.patch, HBASE-12563.patch When I use startMiniHBaseCluster method. It starts up a Mini Cluster for the tests. While starting It changes some configuration. For example Master's port or Region Server's. After Cluster starting We should set its new configuration to HbaseTestingUtils conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11574) hbase:meta's regions can be replicated
[ https://issues.apache.org/jira/browse/HBASE-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-11574: Attachment: meta-replicas-0.98.zip Patch for 0.98 for reference. Will post the master patch soon. hbase:meta's regions can be replicated -- Key: HBASE-11574 URL: https://issues.apache.org/jira/browse/HBASE-11574 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Assignee: Devaraj Das Attachments: meta-replicas-0.98.zip As mentioned elsewhere, we can leverage hbase-10070 features to create replicas for the meta tables regions so that: 1. meta hotspotting can be circumvented 2. meta becomes highly available for reading -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223406#comment-14223406 ] Enis Soztutar commented on HBASE-12128: --- This looks good. TestInterfaceAudienceAnnotations is a new test that checks whether every public class has a InterfaceAudience annotation. I think we should make TableConfiguration a package-protected class so that users do not use it. Also we can annotate it with InterfaceAudience.Private as well. I was thinking about the TableConfiguration and the rpc factories. One alternative approach we can take is to change TableConfiguration to be a TableContext, and have the context own both the configuration and rpc factories (and possibly other table related concept). The patch as it is will do the job. Just a suggestion, it is up to you which one is better. Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12471) Task 4. replace internal ConnectionManager#{delete,get}Connection use with #close, #createConnection (0.98, 0.99) under src/main/java
[
https://issues.apache.org/jira/browse/HBASE-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223416#comment-14223416
]
Hudson commented on HBASE-12471:
SUCCESS: Integrated in HBase-1.0 #497 (See
[https://builds.apache.org/job/HBase-1.0/497/])
HBASE-12471 Task 4. replace internal ConnectionManager#{delete,get}Connection
use with #close, #createConnection (0.98, 0.99) under src/main/java (stack: rev
f3c5f11718fd780f35386305729741afcc5b6241)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController2.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseCluster.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapper.java
* hbase-it/src/test/java/org/apache/hadoop/hbase/DistributedHBaseCluster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionUtils.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionLocator.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKConfig.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHCM.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransactionOnCluster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java
*
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHBaseAdminNoCluster.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationKillRS.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerWithBulkload.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java
HBASE-12471 Task 4. replace internal ConnectionManager#{delete,get}Connection
use with #close, #createConnection (0.98, 0.99) under src/main/java (stack: rev
3cd1d7ab0c04ced7086e4a5010b492af4e46b308)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
Task 4. replace internal ConnectionManager#{delete,get}Connection use with
#close, #createConnection (0.98, 0.99) under src/main/java
-
Key: HBASE-12471
URL: https://issues.apache.org/jira/browse/HBASE-12471
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Fix For: 2.0.0, 0.99.2
Attachments:
0001-HBASE-12471-Task-4.-replace-internal-ConnectionManag.patch, 12404v9.txt,
124471v4.txt, 12471.reapply.reapply.v5.txt, 12471.reapply.reapply.v5.txt,
12471.reapply.reapply.v5.txt, 12471.reapply.reapply.v5.txt,
12471.reapply.reapply.v5.txt, 12471.reapply.txt, 12471.reapply.v4.txt,
12471.reapplyv2.txt, 12471.reapplyv2.txt, 12471.reapplyv3.txt, 12471v10.txt,
12471v10.txt, 12471v11.txt, 12471v2.txt, 12471v3.txt, 12471v6.txt,
12471v7.txt, 12471v9.txt
Let me do this. A bunch of this was done in HBASE-12404 Let me see if can
find more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223417#comment-14223417 ] Hudson commented on HBASE-12550: SUCCESS: Integrated in HBase-1.0 #497 (See [https://builds.apache.org/job/HBase-1.0/497/]) HBASE-12550 Check all storefiles are referenced before splitting (elliott: rev 339fd6bdca12619537b80427abe35f6eea2e4479) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223432#comment-14223432 ] Hudson commented on HBASE-12550: FAILURE: Integrated in HBase-0.98 #697 (See [https://builds.apache.org/job/HBase-0.98/697/]) HBASE-12550 Check all storefiles are referenced before splitting (elliott: rev 860ddd2c22371cf1d5d75f3d94cfad350eafb34f) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12495) Use interfaces in the shell scripts
[
https://issues.apache.org/jira/browse/HBASE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223442#comment-14223442
]
Hadoop QA commented on HBASE-12495:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683372/HBASE-12495B.patch
against master branch at commit 0df5ed2ca6ce3758b4745f63400fd81d17107038.
ATTACHMENT ID: 12683372
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11814//console
This message is automatically generated.
Use interfaces in the shell scripts
---
Key: HBASE-12495
URL: https://issues.apache.org/jira/browse/HBASE-12495
Project: HBase
Issue Type: Bug
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Fix For: 2.0.0, 0.99.2
Attachments: HBASE-12495.patch, HBASE-12495B.patch
Change some explicit calls to HBaseAdmin and HTable to interface counterparts
in the hbase shell scripts
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12548) Improve debuggability of IntegrationTestTimeBoundedRequestsWithRegionReplicas
[ https://issues.apache.org/jira/browse/HBASE-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-12548: Attachment: 12548-0.98.zip Patch for 0.98 for reference. Will post master patch shortly. Improve debuggability of IntegrationTestTimeBoundedRequestsWithRegionReplicas - Key: HBASE-12548 URL: https://issues.apache.org/jira/browse/HBASE-12548 Project: HBase Issue Type: Bug Reporter: Devaraj Das Assignee: Devaraj Das Priority: Minor Fix For: 2.0.0 Attachments: 12548-0.98.zip -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12495) Use interfaces in the shell scripts
[ https://issues.apache.org/jira/browse/HBASE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12495: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed branch-1+. Thanks for the patch [~sduskis] Use interfaces in the shell scripts --- Key: HBASE-12495 URL: https://issues.apache.org/jira/browse/HBASE-12495 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12495.patch, HBASE-12495B.patch Change some explicit calls to HBaseAdmin and HTable to interface counterparts in the hbase shell scripts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223471#comment-14223471 ] stack commented on HBASE-12128: --- Will this work? It caches a tableconfiguration on a Connection instance but a Connection can be used to go against many tables. Should your rather be caching the tableconfiguration in a map keyed by tablename on a Connection? Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12479: --- Comment: was deleted (was: And reopen...) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-12479: Reverted Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223517#comment-14223517 ] Andrew Purtell edited comment on HBASE-12479 at 11/24/14 9:09 PM: -- Reverted from 0.98 was (Author: apurtell): Reverted Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223523#comment-14223523 ] Stephen Yuan Jiang commented on HBASE-12128: [~stack] why not working? The table configuration is the same values that we retrieve for all tables in the connection. If it would be updated by the table instance of a connection, the updated values will stay only with the table; other tables from the Connection instance will not be affected. If using a map keyed by tablename on a Connection, then we have to go through configuration registry again for the same value for a different table. It reduce the improvement from cache. Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12552) listSnapshots should list only owned snapshots for non-super user
[ https://issues.apache.org/jira/browse/HBASE-12552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223524#comment-14223524 ] Andrew Purtell commented on HBASE-12552: This change was only useful for trunk? listSnapshots should list only owned snapshots for non-super user - Key: HBASE-12552 URL: https://issues.apache.org/jira/browse/HBASE-12552 Project: HBase Issue Type: Bug Components: snapshots Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: 12552-v2.patch, HBASE-12552.patch Currently list_snapshots lists all the snapshots available for a non-super user which may not be correct, this should be applicable only for a user with admin rights. So I feel for a non-super user it should lists only the snapshots owned by it if any. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12476) HydraBase Consensus Protocol
[ https://issues.apache.org/jira/browse/HBASE-12476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223528#comment-14223528 ] Rishit Shroff commented on HBASE-12476: --- [~stack]: Thanks for looking through the code and providing the inputs. Please find my responses below: What direction do you think we should take this contrib?Toward the layout described in the hydrabase doc or are you thinking we'd recast it as an in-cluster quorum-of-regions? If the latter, would it be on all the time – a different sort of hbase – or would it be something you could enable? HBase 2.0 or HBase 1.0? bq. Rishit: The RAFT module by itself is pretty independent of in-cluster vs multi-cluster setup. It just deals with peers with ranks and replication. I think the question will be about how we want to take the integration of this protocol into HBase. IMO, we can target for in-cluster mode first and then make it applicable to cross-cluster. Also, since HBase 1.0 is a soon to be released version, I would prefer to put into into HBase 2.0. For sure, we will need to meetup and chat about this. I would prefer to have this as an opt-in option. What should we do about swift out here in apache hbase? We'd redo it as pb and rpc calls? bq. Rishit: The swift jars used in current HBase are old, we definitely need to upgrade them and fix the changed dependencies. airlift is not for REST services, right? You are just using it for stats? You like it? Better than the histograms we have going on out herein apache hbase? You have a Counter, Duration, TimeDistribution, and ExpotentialDecay.. Need airlift for this or could redo doing our current metrics dependencies? bq. Rishit: Yes, even in my profiling, I saw airlift as a hot-spot. I will file a JIRA to move the stats out of airlift and choose the other options (hadoop style, or [~daviddengcn] work for histogram at the very least. Should apache hbase make use of jmxutils too? How you folks using it (Not reviewed the patch yet). bq. Rishit: AFAIK, they are used by airlift package. I will double check. Thanks, Rishit HydraBase Consensus Protocol Key: HBASE-12476 URL: https://issues.apache.org/jira/browse/HBASE-12476 Project: HBase Issue Type: Sub-task Components: Consensus, wal Reporter: Gaurav Menghani Assignee: Gaurav Menghani Attachments: 0001-HydraBase-consensus-protocol.patch This is the first patch for the HydraBase consensus protocol implemented according to the Raft consensus protocol (https://ramcloud.stanford.edu/raft.pdf), as advertised in (HBASE-12259) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12373) Provide a command to list visibility labels
[ https://issues.apache.org/jira/browse/HBASE-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223529#comment-14223529 ] Andrew Purtell commented on HBASE-12373: Better than before [~jerryhe] Provide a command to list visibility labels --- Key: HBASE-12373 URL: https://issues.apache.org/jira/browse/HBASE-12373 Project: HBase Issue Type: Improvement Affects Versions: 0.98.7, 0.99.1 Reporter: Jerry He Assignee: Jerry He Priority: Minor Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12373-master-v2.patch, HBASE-12373-master-v3.patch, HBASE-12373-master.patch A command to list visibility labels that are in place would be handy. This is also in line with many of the other hbase list commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12128: --- Attachment: HBASE-12128.v2-2.0.patch V2 patch fixed the checkstyle issue and unit test failure. Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223559#comment-14223559 ] Hudson commented on HBASE-12550: FAILURE: Integrated in HBase-0.98 #698 (See [https://builds.apache.org/job/HBase-0.98/698/]) HBASE-12550 ADDENDUM Check all storefiles are referenced before splitting (elliott: rev db73050a678de6e50fec5fc413591ea1bac53cf1) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223576#comment-14223576 ] Hudson commented on HBASE-12550: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #665 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/665/]) HBASE-12550 ADDENDUM Check all storefiles are referenced before splitting (elliott: rev db73050a678de6e50fec5fc413591ea1bac53cf1) * hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12495) Use interfaces in the shell scripts
[ https://issues.apache.org/jira/browse/HBASE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223588#comment-14223588 ] Hudson commented on HBASE-12495: SUCCESS: Integrated in HBase-1.0 #498 (See [https://builds.apache.org/job/HBase-1.0/498/]) HBASE-12495 Use interfaces in the shell scripts (solomon duskis) (stack: rev a4a3ffd56094de5fe2a661f7c3f59dff2c9d2b28) * hbase-shell/src/main/ruby/hbase/visibility_labels.rb * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-shell/src/main/ruby/hbase/table.rb * hbase-shell/src/main/ruby/hbase/security.rb * hbase-shell/src/main/ruby/hbase.rb * hbase-shell/src/main/ruby/hbase/admin.rb Use interfaces in the shell scripts --- Key: HBASE-12495 URL: https://issues.apache.org/jira/browse/HBASE-12495 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12495.patch, HBASE-12495B.patch Change some explicit calls to HBaseAdmin and HTable to interface counterparts in the hbase shell scripts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
Scott Fines created HBASE-12565: --- Summary: Race condition in HRegion.batchMutate() causes partial data to be written when region closes Key: HBASE-12565 URL: https://issues.apache.org/jira/browse/HBASE-12565 Project: HBase Issue Type: Bug Components: Performance, regionserver Affects Versions: 0.98.6 Reporter: Scott Fines The following sequence of events is possible to occur in HRegion's batchMutate() call: 1. caller attempts to call HRegion.batchMutate() with a batch of N1 records 2. batchMutate acquires region lock in startRegionOperation, then calls doMiniBatchMutation() 3. doMiniBatchMutation acquires one row lock 4. Region closes 5. doMiniBatchMutation attempts to acquire second row lock. When this happens, the lock acquisition will also attempt to acquire the region lock, which fails (because the region is closing). At this stage, doMiniBatchMutation will stop writing further, BUT it WILL write data for the rows whose locks have already been acquired, and advance the index in MiniBatchOperationInProgress. Then, after it terminates successfully, batchMutate() will loop around a second time, and attempt AGAIN to acquire the region closing lock. When that happens, a NotServingRegionException is thrown back to the caller. Thus, we have a race condition where partial data can be written when a region server is closing. The main problem stems from the location of startRegionOperation() calls in batchMutate and doMiniBatchMutation(): 1. batchMutate() reacquires the region lock with each iteration of the loop, which can cause some successful writes to occur, but then fail on others 2. getRowLock() attempts to acquire the region lock once for each row, which allows doMiniBatchMutation to terminate early; this forces batchMutate() to use multiple iterations and results in condition 1 being hit. There appears to be two parts to the solution as well: 1. open an internal path so that doMiniBatchMutation() can acquire row locks without checking for region closure. This will have the added benefit of a significant performance improvement during large batch mutations. 2. move the startRegionOperation() out of the loop in batchMutate() so that multiple iterations of doMiniBatchMutation will not cause the operation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
[ https://issues.apache.org/jira/browse/HBASE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reassigned HBASE-12565: - Assignee: Vladimir Rodionov Race condition in HRegion.batchMutate() causes partial data to be written when region closes - Key: HBASE-12565 URL: https://issues.apache.org/jira/browse/HBASE-12565 Project: HBase Issue Type: Bug Components: Performance, regionserver Affects Versions: 0.98.6 Reporter: Scott Fines Assignee: Vladimir Rodionov The following sequence of events is possible to occur in HRegion's batchMutate() call: 1. caller attempts to call HRegion.batchMutate() with a batch of N1 records 2. batchMutate acquires region lock in startRegionOperation, then calls doMiniBatchMutation() 3. doMiniBatchMutation acquires one row lock 4. Region closes 5. doMiniBatchMutation attempts to acquire second row lock. When this happens, the lock acquisition will also attempt to acquire the region lock, which fails (because the region is closing). At this stage, doMiniBatchMutation will stop writing further, BUT it WILL write data for the rows whose locks have already been acquired, and advance the index in MiniBatchOperationInProgress. Then, after it terminates successfully, batchMutate() will loop around a second time, and attempt AGAIN to acquire the region closing lock. When that happens, a NotServingRegionException is thrown back to the caller. Thus, we have a race condition where partial data can be written when a region server is closing. The main problem stems from the location of startRegionOperation() calls in batchMutate and doMiniBatchMutation(): 1. batchMutate() reacquires the region lock with each iteration of the loop, which can cause some successful writes to occur, but then fail on others 2. getRowLock() attempts to acquire the region lock once for each row, which allows doMiniBatchMutation to terminate early; this forces batchMutate() to use multiple iterations and results in condition 1 being hit. There appears to be two parts to the solution as well: 1. open an internal path so that doMiniBatchMutation() can acquire row locks without checking for region closure. This will have the added benefit of a significant performance improvement during large batch mutations. 2. move the startRegionOperation() out of the loop in batchMutate() so that multiple iterations of doMiniBatchMutation will not cause the operation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12495) Use interfaces in the shell scripts
[ https://issues.apache.org/jira/browse/HBASE-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223638#comment-14223638 ] Hudson commented on HBASE-12495: SUCCESS: Integrated in HBase-TRUNK #5813 (See [https://builds.apache.org/job/HBase-TRUNK/5813/]) HBASE-12495 Use interfaces in the shell scripts (solomon duskis) (stack: rev 7893c013bc4902a5b0e4ea4371aa9232df968578) * hbase-shell/src/main/ruby/hbase/quotas.rb * hbase-shell/src/main/ruby/hbase/admin.rb * hbase-shell/src/main/ruby/hbase/visibility_labels.rb * hbase-shell/src/main/ruby/hbase/table.rb * hbase-shell/src/test/java/org/apache/hadoop/hbase/client/TestShell.java * hbase-shell/src/main/ruby/hbase.rb * hbase-shell/src/main/ruby/hbase/security.rb Use interfaces in the shell scripts --- Key: HBASE-12495 URL: https://issues.apache.org/jira/browse/HBASE-12495 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 0.99.2 Attachments: HBASE-12495.patch, HBASE-12495B.patch Change some explicit calls to HBaseAdmin and HTable to interface counterparts in the hbase shell scripts -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12565) Race condition in HRegion.batchMutate() causes partial data to be written when region closes
[ https://issues.apache.org/jira/browse/HBASE-12565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reassigned HBASE-12565: - Assignee: (was: Vladimir Rodionov) Race condition in HRegion.batchMutate() causes partial data to be written when region closes - Key: HBASE-12565 URL: https://issues.apache.org/jira/browse/HBASE-12565 Project: HBase Issue Type: Bug Components: Performance, regionserver Affects Versions: 0.98.6 Reporter: Scott Fines The following sequence of events is possible to occur in HRegion's batchMutate() call: 1. caller attempts to call HRegion.batchMutate() with a batch of N1 records 2. batchMutate acquires region lock in startRegionOperation, then calls doMiniBatchMutation() 3. doMiniBatchMutation acquires one row lock 4. Region closes 5. doMiniBatchMutation attempts to acquire second row lock. When this happens, the lock acquisition will also attempt to acquire the region lock, which fails (because the region is closing). At this stage, doMiniBatchMutation will stop writing further, BUT it WILL write data for the rows whose locks have already been acquired, and advance the index in MiniBatchOperationInProgress. Then, after it terminates successfully, batchMutate() will loop around a second time, and attempt AGAIN to acquire the region closing lock. When that happens, a NotServingRegionException is thrown back to the caller. Thus, we have a race condition where partial data can be written when a region server is closing. The main problem stems from the location of startRegionOperation() calls in batchMutate and doMiniBatchMutation(): 1. batchMutate() reacquires the region lock with each iteration of the loop, which can cause some successful writes to occur, but then fail on others 2. getRowLock() attempts to acquire the region lock once for each row, which allows doMiniBatchMutation to terminate early; this forces batchMutate() to use multiple iterations and results in condition 1 being hit. There appears to be two parts to the solution as well: 1. open an internal path so that doMiniBatchMutation() can acquire row locks without checking for region closure. This will have the added benefit of a significant performance improvement during large batch mutations. 2. move the startRegionOperation() out of the loop in batchMutate() so that multiple iterations of doMiniBatchMutation will not cause the operation to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12550) Check all storefiles are referenced before splitting
[
https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-12550:
This change breaks Apache Phoenix compilation:
{code}
[ERROR]
/Users/apurtell/src/phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java:[364,28]
method createDaughterRegionFromSplits in class
org.apache.hadoop.hbase.regionserver.HRegion cannot be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.HRegionInfo,int
[ERROR] found: org.apache.hadoop.hbase.HRegionInfo
[ERROR] reason: actual and formal argument lists differ in length
[ERROR]
/Users/apurtell/src/phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java:[368,28]
method createDaughterRegionFromSplits in class
org.apache.hadoop.hbase.regionserver.HRegion cannot be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.HRegionInfo,int
[ERROR] found: org.apache.hadoop.hbase.HRegionInfo
[ERROR] reason: actual and formal argument lists differ in length
{code}
Can we get an addendum for 0.98 that fixes that?
Check all storefiles are referenced before splitting
Key: HBASE-12550
URL: https://issues.apache.org/jira/browse/HBASE-12550
Project: HBase
Issue Type: Bug
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch,
HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch,
HBASE-12550.patch
Make double extra sure all storefiles are referenced before
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-12550) Check all storefiles are referenced before splitting
[
https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223652#comment-14223652
]
Andrew Purtell edited comment on HBASE-12550 at 11/24/14 10:32 PM:
---
This change breaks Apache Phoenix compilation:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
{noformat}
[ERROR]
/Users/apurtell/src/phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java:[364,28]
method createDaughterRegionFromSplits in class
org.apache.hadoop.hbase.regionserver.HRegion cannot be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.HRegionInfo,int
[ERROR] found: org.apache.hadoop.hbase.HRegionInfo
[ERROR] reason: actual and formal argument lists differ in length
[ERROR]
/Users/apurtell/src/phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java:[368,28]
method createDaughterRegionFromSplits in class
org.apache.hadoop.hbase.regionserver.HRegion cannot be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.HRegionInfo,int
[ERROR] found: org.apache.hadoop.hbase.HRegionInfo
[ERROR] reason: actual and formal argument lists differ in length
{noformat}
Can we get an addendum for 0.98 that fixes that?
was (Author: apurtell):
This change breaks Apache Phoenix compilation:
{code}
[ERROR]
/Users/apurtell/src/phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java:[364,28]
method createDaughterRegionFromSplits in class
org.apache.hadoop.hbase.regionserver.HRegion cannot be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.HRegionInfo,int
[ERROR] found: org.apache.hadoop.hbase.HRegionInfo
[ERROR] reason: actual and formal argument lists differ in length
[ERROR]
/Users/apurtell/src/phoenix/phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexSplitTransaction.java:[368,28]
method createDaughterRegionFromSplits in class
org.apache.hadoop.hbase.regionserver.HRegion cannot be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.HRegionInfo,int
[ERROR] found: org.apache.hadoop.hbase.HRegionInfo
[ERROR] reason: actual and formal argument lists differ in length
{code}
Can we get an addendum for 0.98 that fixes that?
Check all storefiles are referenced before splitting
Key: HBASE-12550
URL: https://issues.apache.org/jira/browse/HBASE-12550
Project: HBase
Issue Type: Bug
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch,
HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch,
HBASE-12550.patch
Make double extra sure all storefiles are referenced before
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
Andrew Purtell created HBASE-12566:
--
Summary: HRegion should have an InterfaceAudience of
LimitedPrivate(PHOENIX)
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has an IndexSplitTransaction
class that was broken by a change to a package scoped method:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223692#comment-14223692 ] Elliott Clark commented on HBASE-12550: --- HRegion is @Private. I thought it was ok to change anything that's @private Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223713#comment-14223713 ] Andrew Purtell commented on HBASE-12550: I filed HBASE-12566 a few minutes ago now that we're aware the audience annotation on HRegion doesn't reflect what Phoenix is doing. Setting aside the question if they should have depended on this method or not, every breaking change like this limits what HBase versions a Phoenix release can support. It would be a big help if we could help them ride over this change, with a backwards compatible method or similar. Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12566:
---
Description:
I've discovered after HBASE-12550 that Phoenix has a class that was broken by a
change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
was:
I've discovered after HBASE-12550 that Phoenix has an IndexSplitTransaction
class that was broken by a change to a package scoped method:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223723#comment-14223723 ] Elliott Clark commented on HBASE-12550: --- So we can provide something that's backwards compatible. However lets continue the annotation discussion on the other jira. Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223732#comment-14223732
]
Elliott Clark commented on HBASE-12566:
---
If HRegion isn't private I don't really see anything being private.
This is a pretty slippery slope to allow other code to reach so far in.
Creating splits is something that requires coordination with other parts of the
system and can cause severe corruption if not handled correctly.
HBase is on the hook for making sure that things are stored correctly. We
shouldn't expose anything that can cause us to lose data.
Additionally code that ignore the annotations telling who can access what do so
at their own risk; not with the knowledge that we'll open everything up at the
first asking. That's a sure road to making everything public and making it so
nothing can be changed.
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223737#comment-14223737
]
Elliott Clark commented on HBASE-12566:
---
Additionally this is a package private that's being used by an external
project. Something is majorly wrong here
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223740#comment-14223740
]
Sean Busbey commented on HBASE-12566:
-
HRegion is pretty internal and I'm not sure we should be opening it up more,
even to LimitedPrivate. According to our soon-to-be compat guide, doing this
would mean we wouldn't change it within a minor version, which is a big
limitation. I get that through the CoprocessorEnvironment coprocessors can get
some internal objects that we list as IA.Private. I've always taken the
explicit scoping of those internals (like HRegoin and WAL) to mean that
implementers were in hairy territory and should take care.
Additionally, IA.LimitedPrivate notation wouldn't have stopped HBASE-12550 from
breaking Phoenix since that change was to a package-scoped method.
Could we instead modify an already LimitedPrivate(PHOENIX) api somewhere else?
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[
https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223743#comment-14223743
]
Hadoop QA commented on HBASE-12128:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683418/HBASE-12128.v2-2.0.patch
against master branch at commit 7893c013bc4902a5b0e4ea4371aa9232df968578.
ATTACHMENT ID: 12683418
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11816//console
This message is automatically generated.
Cache configuration and RpcController selection for Table in Connection
---
Key: HBASE-12128
URL: https://issues.apache.org/jira/browse/HBASE-12128
Project: HBase
Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Andrew Purtell
Assignee: Stephen Yuan Jiang
Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2
Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch
Original Estimate: 120h
Time Spent: 72h
Remaining Estimate: 48h
Creating Table instances should be lightweight. Apps that manage their own
Connections are expected to create Tables on demand for each interaction.
However we look up values from Hadoop Configuration when constructing Table
objects for storing to some of its fields. Configuration is a heavyweight
registry that does a lot of string operations and regex matching. Method
calls into Configuration account for 48.25% of CPU time when creating the
HTable object in 0.98. Another ~48% of CPU is spent constructing the desired
RpcController object via reflection in 0.98. Together this can
[jira] [Commented] (HBASE-11689) Track meta in transition
[ https://issues.apache.org/jira/browse/HBASE-11689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223747#comment-14223747 ] Hudson commented on HBASE-11689: SUCCESS: Integrated in HBase-0.98 #699 (See [https://builds.apache.org/job/HBase-0.98/699/]) Revert HBASE-12479 Backport HBASE-11689 (Track meta in transition) (Virag Kothari) (apurtell: rev 11bd497a9b33f47fa632f0fd06c758f0fdb79d20) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java * hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaRegionTracker.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-protocol/src/main/protobuf/ZooKeeper.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java Track meta in transition Key: HBASE-11689 URL: https://issues.apache.org/jira/browse/HBASE-11689 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 1.0.0, 2.0.0 Reporter: Jimmy Xiang Assignee: Andrey Stepachev Fix For: 2.0.0 Attachments: HBASE-11689-branch-1.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, hbase-11689-revised.patch, hbase-11689-revised_2.patch, hbase-11689-revised_2.patch With ZK-less region assignment, user regions in transition are tracked in meta. We need a way to track meta in transition too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223749#comment-14223749
]
Sean Busbey commented on HBASE-12566:
-
As an example that isn't Phoenix, the HBase Lily Indexer is going to get broken
pretty hard by the recent WAL refactoring. Along with nearly every part of a
WAL (HLogKey, WALEdit, etc), they were referencing HLogUtil. Should we have
expanded LP annotations to cover their use?
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223746#comment-14223746 ] Hudson commented on HBASE-12479: SUCCESS: Integrated in HBase-0.98 #699 (See [https://builds.apache.org/job/HBase-0.98/699/]) Revert HBASE-12479 Backport HBASE-11689 (Track meta in transition) (Virag Kothari) (apurtell: rev 11bd497a9b33f47fa632f0fd06c758f0fdb79d20) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java * hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaRegionTracker.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-protocol/src/main/protobuf/ZooKeeper.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12128: --- Status: Patch Available (was: In Progress) Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12128: --- Status: In Progress (was: Patch Available) Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223759#comment-14223759 ] stack commented on HBASE-12128: --- I see. Then suggest renaming the configuration 'cache'; call it ConfigurationSnapshot or CachedConfiguration? And why not have Admin and RegionLocators also use this cached Configuration? Thanks for working on this. Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223760#comment-14223760
]
Andrew Purtell commented on HBASE-12566:
See my comment on PHOENIX-1479
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-12550: -- Attachment: 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch So here's a patch that can get Phoenix up and working while they try and get out of the splitting code. Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-12550: -- Status: Patch Available (was: Reopened) Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223798#comment-14223798 ] Andrew Purtell commented on HBASE-12550: Thanks, +1 on the addendum for 0.98. Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11689) Track meta in transition
[ https://issues.apache.org/jira/browse/HBASE-11689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223808#comment-14223808 ] Hudson commented on HBASE-11689: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #666 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/666/]) Revert HBASE-12479 Backport HBASE-11689 (Track meta in transition) (Virag Kothari) (apurtell: rev 11bd497a9b33f47fa632f0fd06c758f0fdb79d20) * hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaRegionTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-protocol/src/main/protobuf/ZooKeeper.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Track meta in transition Key: HBASE-11689 URL: https://issues.apache.org/jira/browse/HBASE-11689 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 1.0.0, 2.0.0 Reporter: Jimmy Xiang Assignee: Andrey Stepachev Fix For: 2.0.0 Attachments: HBASE-11689-branch-1.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, HBASE-11689.patch, hbase-11689-revised.patch, hbase-11689-revised_2.patch, hbase-11689-revised_2.patch With ZK-less region assignment, user regions in transition are tracked in meta. We need a way to track meta in transition too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12479) Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1
[ https://issues.apache.org/jira/browse/HBASE-12479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223807#comment-14223807 ] Hudson commented on HBASE-12479: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #666 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/666/]) Revert HBASE-12479 Backport HBASE-11689 (Track meta in transition) (Virag Kothari) (apurtell: rev 11bd497a9b33f47fa632f0fd06c758f0fdb79d20) * hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java * hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java * hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterFailover.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaRegionTracker.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * hbase-protocol/src/main/protobuf/ZooKeeper.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Backport HBASE-11689 (Track meta in transition) to 0.98 and branch-1 Key: HBASE-12479 URL: https://issues.apache.org/jira/browse/HBASE-12479 Project: HBase Issue Type: Bug Components: Region Assignment Reporter: Virag Kothari Assignee: Virag Kothari Fix For: 0.98.9, 0.99.2 Attachments: HBASE-12479-0.98.patch, HBASE-12479-0.98_v2.patch, HBASE-12479-branch-1.patch Required for zk-less assignment -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223833#comment-14223833 ] Enis Soztutar commented on HBASE-12128: --- The configurations parsed from conf object is specific to HTable instance for now. So it is kind of tied to the HTable internals (write buffer size, etc). I think it is not a generic configuration cache or anything, just a way for faster HTable construction. Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223835#comment-14223835 ] Enis Soztutar commented on HBASE-6721: -- I left some comments in RB, but I wanted to comment on high level things here. I think this is generally a useful feature, since we are seeing more and more multi-tenant use cases, and without really good isolation rs groups allow the cluster to be shared more easily. - I was reading up on the recent YARN label feature (YARN-796), since the ideas are very similar. However, in YARN labels, a node can have multiple labels, versus here, a server can only have one group. Will it be better to have one-to-many relationship from servers to groups? All servers will have the default group, as well as any more groups that it is assigned. We can do the same thing for tables as well. By default, tables will belong to group default, but can be added more. For assignment, we just look at all the cumulative list of servers that this region will be assignable to. Will this complicate assignment? - Alternatively we can also name this labels (just a suggestion). - For all new features, I think it is fair to request one single source of truth and also some more transactional guarantees for operations. If possible, we should not use zk at all (for caching as done in patch). Instead we can just open the region from master. This is different than the co-location discussion for master and meta. This table is tiny, and not query'able from clients. The data is just there for the master to access. If we do this, we do not even have to have a cache. - I am ok with bringing this as a core functionality rather than CP + LB. The reasoning is that as clusters grow bigger, more users might be interested in this for better isolation. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Attachments: 6721-master-webUI.patch, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721-DesigDoc.pdf, HBASE-6721_10.patch, HBASE-6721_8.patch, HBASE-6721_9.patch, HBASE-6721_9.patch, HBASE-6721_94.patch, HBASE-6721_94.patch, HBASE-6721_94_2.patch, HBASE-6721_94_3.patch, HBASE-6721_94_3.patch, HBASE-6721_94_4.patch, HBASE-6721_94_5.patch, HBASE-6721_94_6.patch, HBASE-6721_94_7.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk.patch, HBASE-6721_trunk1.patch, HBASE-6721_trunk2.patch In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12566) HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
[
https://issues.apache.org/jira/browse/HBASE-12566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Taylor updated HBASE-12566:
-
Labels: Phoenix (was: )
HRegion should have an InterfaceAudience of LimitedPrivate(PHOENIX)
---
Key: HBASE-12566
URL: https://issues.apache.org/jira/browse/HBASE-12566
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Labels: Phoenix
Fix For: 2.0.0, 0.98.9, 0.99.2
I've discovered after HBASE-12550 that Phoenix has a class that was broken by
a change to a package scoped method in HRegion:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/reg
index 39a9fdc..3377e6b 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -4628,11 +4628,12 @@ public class HRegion implements HeapSize { // ,
Writable{
/**
* Create a daughter region from given a temp directory with the region
data.
* @param hri Spec. for daughter region to open.
+ * @param expectedReferenceFileCount
* @throws IOException
*/
- HRegion createDaughterRegionFromSplits(final HRegionInfo hri) throws
IOException {
+ HRegion createDaughterRegionFromSplits(final HRegionInfo hri, int
expectedReferenceFileCount) throws IOException {
// Move the files from the temporary .splits to the final /table/region
directory
-fs.commitDaughterRegion(hri);
+fs.commitDaughterRegion(hri, expectedReferenceFileCount);
{code}
We should change the HRegion InterfaceAudience to LimitedPrivate(COPROC,
PHOENIX).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12559) Provide LoadBalancer with online configuration capability
[ https://issues.apache.org/jira/browse/HBASE-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223850#comment-14223850 ] Ted Yu commented on HBASE-12559: Configuration doesn't provide method to persist changes made to an instance of Configuration. For testing, a subclass of Configuration needs to persist changes to the hbase-site.xml which is on the classpath. Not sure if the above approach is desirable. Provide LoadBalancer with online configuration capability - Key: HBASE-12559 URL: https://issues.apache.org/jira/browse/HBASE-12559 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.0.0 Attachments: 12559-v1.txt, 12559-v2.txt StochasticLoadBalancer has many knobs which user can adjust. It would increase productivity by allowing StochasticLoadBalancer to accept online configuration changes. LoadBalancer already implements setConf(Configuration) method which reloads relevant configuration parameters. We need to add updateMasterConfiguration() method to Admin which invokes the setConf() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10483) Provide API for retrieving info port when hbase.master.info.port is set to 0
[ https://issues.apache.org/jira/browse/HBASE-10483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223849#comment-14223849 ] Enis Soztutar commented on HBASE-10483: --- Sorry for coming to this later. I was going over the methods in Admin, and bumped into this. I think we should not have the method Admin.getMasterInfoPort(). The reasoning is that, that method will only return the active master's info port, but not any other detail about the active master. So you have to obtain the hostname of the active master one way, and obtain the info port some other way. That means when the master is failing over etc, it can go out of sync. For region server info ports, we did a hacky-ish way of adding the info port to the ServerLoad object. Since master and RS are the same daemon now, their info port is the same? Can we create a follow up jira to either change the method to return the full information for the active master or remove the method and rely on ClusterStatus ? Provide API for retrieving info port when hbase.master.info.port is set to 0 Key: HBASE-10483 URL: https://issues.apache.org/jira/browse/HBASE-10483 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Liu Shaohui Fix For: 2.0.0, 0.99.2 Attachments: HBASE-10483-trunk-v1.diff, HBASE-10483-trunk-v2.diff, HBASE-10483-v3.diff, HBASE-10483-v4.diff, HBASE-10483-v5.diff When hbase.master.info.port is set to 0, info port is dynamically determined. An API should be provided so that client can retrieve the actual info port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12567) Track remaining issues for HBase-1.0
Enis Soztutar created HBASE-12567: - Summary: Track remaining issues for HBase-1.0 Key: HBASE-12567 URL: https://issues.apache.org/jira/browse/HBASE-12567 Project: HBase Issue Type: Task Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.2 Just for tracking (a couple of) remaining jiras for 1.0. This will track only absolute criticals. I am thinking that 0.99.2 will be the last 0.99 release. We should do 1.0.0RC shortly afterwards. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12567) Track remaining issues for HBase-1.0
[ https://issues.apache.org/jira/browse/HBASE-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223868#comment-14223868 ] Enis Soztutar commented on HBASE-12567: --- I've created the 1.0.1 and 1.1.0 jira labels. Please mark open issues with that, if you want to target 1.x releases. Track remaining issues for HBase-1.0 Key: HBASE-12567 URL: https://issues.apache.org/jira/browse/HBASE-12567 Project: HBase Issue Type: Task Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.2 Just for tracking (a couple of) remaining jiras for 1.0. This will track only absolute criticals. I am thinking that 0.99.2 will be the last 0.99 release. We should do 1.0.0RC shortly afterwards. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12128) Cache configuration and RpcController selection for Table in Connection
[ https://issues.apache.org/jira/browse/HBASE-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223871#comment-14223871 ] stack commented on HBASE-12128: --- bq. ... for now We don't foresee wanting to cache configs for Admin that we get off the same Connection any time soon? Looking in constructor for HBaseAdmin, it wants to use some of the configs you are caching in this patch (RegionLocator is currently an HTable instance so it will get the benefit this patch adds) Anyways, just a suggestion. +1 on the patch even as is. Cache configuration and RpcController selection for Table in Connection --- Key: HBASE-12128 URL: https://issues.apache.org/jira/browse/HBASE-12128 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Stephen Yuan Jiang Fix For: 1.0.0, 2.0.0, 0.98.9, 0.99.2 Attachments: HBASE-12128.v1-2.0.patch, HBASE-12128.v2-2.0.patch Original Estimate: 120h Time Spent: 72h Remaining Estimate: 48h Creating Table instances should be lightweight. Apps that manage their own Connections are expected to create Tables on demand for each interaction. However we look up values from Hadoop Configuration when constructing Table objects for storing to some of its fields. Configuration is a heavyweight registry that does a lot of string operations and regex matching. Method calls into Configuration account for 48.25% of CPU time when creating the HTable object in 0.98. Another ~48% of CPU is spent constructing the desired RpcController object via reflection in 0.98. Together this can account for ~20% of total on-CPU time of the client. See parent issue for more detail. We are using Connection like a factory for Table. We should cache configuration for Table in Connection. We should also create by reflection once and cache the desired RpcController object, and clone it for new Tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12567) Track remaining issues for HBase-1.0
[ https://issues.apache.org/jira/browse/HBASE-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223872#comment-14223872 ] Enis Soztutar commented on HBASE-12567: --- Linking HBASE-12522. It should be included since major change applicable for 1.x. Track remaining issues for HBase-1.0 Key: HBASE-12567 URL: https://issues.apache.org/jira/browse/HBASE-12567 Project: HBase Issue Type: Task Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.2 Just for tracking (a couple of) remaining jiras for 1.0. This will track only absolute criticals. I am thinking that 0.99.2 will be the last 0.99 release. We should do 1.0.0RC shortly afterwards. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12568) Adopt Semantic Versioning and document it in the book
Enis Soztutar created HBASE-12568: - Summary: Adopt Semantic Versioning and document it in the book Key: HBASE-12568 URL: https://issues.apache.org/jira/browse/HBASE-12568 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Fix For: 0.99.2 See http://search-hadoop.com/m/DHED4LFNzP/semantic+versioningsubj=Re+HBase+Semantic+Versioning We should put that in the book. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12567) Track remaining issues for HBase-1.0
[ https://issues.apache.org/jira/browse/HBASE-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223882#comment-14223882 ] Enis Soztutar commented on HBASE-12567: --- Finish up InterfaceAudience work from HBASE-10462. Track remaining issues for HBase-1.0 Key: HBASE-12567 URL: https://issues.apache.org/jira/browse/HBASE-12567 Project: HBase Issue Type: Task Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.99.2 Just for tracking (a couple of) remaining jiras for 1.0. This will track only absolute criticals. I am thinking that 0.99.2 will be the last 0.99 release. We should do 1.0.0RC shortly afterwards. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-12550: -- Resolution: Fixed Status: Resolved (was: Patch Available) Re-resolving. I just pushed this to 0.98+ Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[
https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223911#comment-14223911
]
Hadoop QA commented on HBASE-12550:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683444/0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch
against master branch at commit 7893c013bc4902a5b0e4ea4371aa9232df968578.
ATTACHMENT ID: 12683444
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11817//console
This message is automatically generated.
Check all storefiles are referenced before splitting
Key: HBASE-12550
URL: https://issues.apache.org/jira/browse/HBASE-12550
Project: HBase
Issue Type: Bug
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch,
0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch,
HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch,
HBASE-12550.patch
Make double extra sure all storefiles are referenced before
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12569) Control MaxDirectMemorySize in the same manner as heap size
Patrick White created HBASE-12569: - Summary: Control MaxDirectMemorySize in the same manner as heap size Key: HBASE-12569 URL: https://issues.apache.org/jira/browse/HBASE-12569 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.98.7 Environment: CentOS 6 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.9, 0.99.2 It would make it much easier on us if we could manage MaxDirectMemorySize in the same way as heap. This patch allows that to happen. Note: I have not tested the *.cmd modifications as I don't have a Windows box (at the moment, can test at home) but look like they should work (famous last words). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12569) Control MaxDirectMemorySize in the same manner as heap size
[ https://issues.apache.org/jira/browse/HBASE-12569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick White updated HBASE-12569: -- Status: Patch Available (was: Open) Control MaxDirectMemorySize in the same manner as heap size --- Key: HBASE-12569 URL: https://issues.apache.org/jira/browse/HBASE-12569 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.98.7 Environment: CentOS 6 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12569-Update-scripts-to-control-MaxDirectMemor.patch It would make it much easier on us if we could manage MaxDirectMemorySize in the same way as heap. This patch allows that to happen. Note: I have not tested the *.cmd modifications as I don't have a Windows box (at the moment, can test at home) but look like they should work (famous last words). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12569) Control MaxDirectMemorySize in the same manner as heap size
[ https://issues.apache.org/jira/browse/HBASE-12569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick White updated HBASE-12569: -- Attachment: 0001-HBASE-12569-Update-scripts-to-control-MaxDirectMemor.patch Control MaxDirectMemorySize in the same manner as heap size --- Key: HBASE-12569 URL: https://issues.apache.org/jira/browse/HBASE-12569 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.98.7 Environment: CentOS 6 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12569-Update-scripts-to-control-MaxDirectMemor.patch It would make it much easier on us if we could manage MaxDirectMemorySize in the same way as heap. This patch allows that to happen. Note: I have not tested the *.cmd modifications as I don't have a Windows box (at the moment, can test at home) but look like they should work (famous last words). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12570) Missing/invalid split policy class name brings down your HBase cluster
James Taylor created HBASE-12570: Summary: Missing/invalid split policy class name brings down your HBase cluster Key: HBASE-12570 URL: https://issues.apache.org/jira/browse/HBASE-12570 Project: HBase Issue Type: Bug Reporter: James Taylor See PHOENIX-1473. If a split policy class cannot be resolved, then your HBase cluster will be brought down as each region server that successively attempts to open the region will not find the class and will bring itself down. One idea to prevent this would be to fail the CREATE TABLE or ALTER TABLE admin call if the split policy class cannot be found. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12569) Control MaxDirectMemorySize in the same manner as heap size
[ https://issues.apache.org/jira/browse/HBASE-12569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223958#comment-14223958 ] stack commented on HBASE-12569: --- +1 Control MaxDirectMemorySize in the same manner as heap size --- Key: HBASE-12569 URL: https://issues.apache.org/jira/browse/HBASE-12569 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.98.7 Environment: CentOS 6 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12569-Update-scripts-to-control-MaxDirectMemor.patch It would make it much easier on us if we could manage MaxDirectMemorySize in the same way as heap. This patch allows that to happen. Note: I have not tested the *.cmd modifications as I don't have a Windows box (at the moment, can test at home) but look like they should work (famous last words). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12404) Task 5 from parent: Replace internal HTable constructor use with HConnection#getTable (0.98, 0.99)
[ https://issues.apache.org/jira/browse/HBASE-12404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12404: -- Attachment: 12404v19.txt Fix a few more tests Task 5 from parent: Replace internal HTable constructor use with HConnection#getTable (0.98, 0.99) -- Key: HBASE-12404 URL: https://issues.apache.org/jira/browse/HBASE-12404 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Fix For: 0.99.2 Attachments: 0001-HBASE-12404-Task-5-from-parent-Replace-internal-HTab.patch, 12404.txt, 12404.v16.txt, 12404.v16.txt, 12404.v17.txt, 12404v10.txt, 12404v11.txt, 12404v12.txt, 12404v12.txt, 12404v13.txt, 12404v13.txt, 12404v14.txt, 12404v15.txt, 12404v18.txt, 12404v19.txt, 12404v2.txt, 12404v3.txt, 12404v5.txt, 12404v6.txt, 12404v7.txt, 12404v9.txt Do the step 5. from the [~ndimiduk] list in parent issue. Go through src code and change all new HTable to instead be connection.getTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12570) Missing/invalid split policy class name brings down your HBase cluster
[
https://issues.apache.org/jira/browse/HBASE-12570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223968#comment-14223968
]
Enis Soztutar commented on HBASE-12570:
---
This is from HMaster.sanityCheckTableDescriptor() method introduced in
HBASE-10591
{code}
// check split policy class can be loaded
try {
RegionSplitPolicy.getSplitPolicyClass(htd, conf);
} catch (Exception ex) {
throw new DoNotRetryIOException(ex);
}
{code}
So it should have failed the alter table or create table. Which version of
HBase was the user running? Maybe it was disabled via configuration.
However, we load other classes (not just split policy), and they will have the
same affect as well. For example we have seen a similar problem for
co-processors ([~ndimiduk] was involved in a case where a typo in co-processor
name caused the cluster to go down). Similarly storage engine classes, wal
readers etc.
Missing/invalid split policy class name brings down your HBase cluster
--
Key: HBASE-12570
URL: https://issues.apache.org/jira/browse/HBASE-12570
Project: HBase
Issue Type: Bug
Reporter: James Taylor
See PHOENIX-1473. If a split policy class cannot be resolved, then your HBase
cluster will be brought down as each region server that successively attempts
to open the region will not find the class and will bring itself down.
One idea to prevent this would be to fail the CREATE TABLE or ALTER TABLE
admin call if the split policy class cannot be found.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12404) Task 5 from parent: Replace internal HTable constructor use with HConnection#getTable (0.98, 0.99)
[
https://issues.apache.org/jira/browse/HBASE-12404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223971#comment-14223971
]
Hadoop QA commented on HBASE-12404:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12683465/12404v19.txt
against master branch at commit e83082a88816684714d8a563967046e582f9b8c7.
ATTACHMENT ID: 12683465
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 133
new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11819//console
This message is automatically generated.
Task 5 from parent: Replace internal HTable constructor use with
HConnection#getTable (0.98, 0.99)
--
Key: HBASE-12404
URL: https://issues.apache.org/jira/browse/HBASE-12404
Project: HBase
Issue Type: Sub-task
Reporter: stack
Assignee: stack
Fix For: 0.99.2
Attachments:
0001-HBASE-12404-Task-5-from-parent-Replace-internal-HTab.patch, 12404.txt,
12404.v16.txt, 12404.v16.txt, 12404.v17.txt, 12404v10.txt, 12404v11.txt,
12404v12.txt, 12404v12.txt, 12404v13.txt, 12404v13.txt, 12404v14.txt,
12404v15.txt, 12404v18.txt, 12404v19.txt, 12404v2.txt, 12404v3.txt,
12404v5.txt, 12404v6.txt, 12404v7.txt, 12404v9.txt
Do the step 5. from the [~ndimiduk] list in parent issue. Go through src
code and change all new HTable to instead be connection.getTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223974#comment-14223974 ] Hudson commented on HBASE-12550: SUCCESS: Integrated in HBase-1.0 #499 (See [https://builds.apache.org/job/HBase-1.0/499/]) HBASE-12550 ADDENDUM Make HRegion's api not change. (elliott: rev 524dcd13234707adfb0a774415b44e6566ad8128) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11902) RegionServer was blocked while aborting
[
https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223976#comment-14223976
]
Qiang Tian commented on HBASE-11902:
from the stacktrace, DrainBarrier.stopAndDrainOps is waiting, so
DrainBarrier#endOp does not notify it.
looking at class DrainBarrier, it is expected that beginOp and endOp are called
in pair. the initial value of {{valueAndFlags}} is 2, incremented by 2 in
beginOp; decremented by 2 in endOp.
in stopAndDrainOps, if getValue(oldValAndFlags) == 1, means oldValAndFlags=2,
all ops are completed in pair, otherwise, it needs to wait the last endOp to
notify it:
{code}
if (getValue(oldValAndFlags) == 1) return; // There were no operations
outstanding.
synchronized (this) { this.wait(); }
{code}
so the problem could be the beginOp/endOp is not called in pair, the hole looks
to be here:
HRegion#internalFlushcache
{code}
// sync unflushed WAL changes when deferred log sync is enabled
// see HBASE-8208 for details
if (wal != null !shouldSyncLog()) {
wal.sync();
}
{code}
at that point, wal.startCacheFlush-closeBarrier.beginOp is called, but
completeCacheFlush-closeBarrier.endOp() is not protected by a try block..so if
WAL/HDFS layer throws exception, the endOp will not be called.
related info in log:
{quote}
2014-09-03 13:38:03,789 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog:
Error while AsyncWriter write, request close of hlog
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,789 FATAL org.apache.hadoop.hbase.regionserver.wal.FSHLog:
Error while AsyncSyncer sync, request close of hlog
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,799 ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush failed for
region page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c.
//==MemStoreFlusher#flushRegion
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
{quote}
the exception thrown to caller should be here:
{code}
FSHLog#syncer:
if (txid = this.failedTxid.get()) {
assert asyncIOE != null :
current txid is among(under) failed txids, but asyncIOE is null!;
throw asyncIOE;
}
{code}
the master branch can catch the hdfs exception, but it just ignore it, which
looks incorrect:
{code}
if (wal != null) {
try {
wal.sync(); // ensure that flush marker is sync'ed
} catch (IOException ioe) {
LOG.warn(Unexpected exception while wal.sync(), ignoring. Exception:
+ StringUtils.stringifyException(ioe));
}
}
{code}
Personally the exeception should not be ignored since it is severe hdfs error.
RegionServer was blocked while aborting
---
Key: HBASE-11902
URL: https://issues.apache.org/jira/browse/HBASE-11902
Project: HBase
Issue Type: Bug
Components: regionserver, wal
Affects Versions: 0.98.4
Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7
Reporter: Victor Xu
Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log,
jstack_hadoop461.cm6.log
Generally, regionserver automatically aborts when isHealth() returns false.
But it sometimes got blocked while aborting. I saved the jstack and logs, and
found out that it was caused by datanodes failures. The regionserver60020
thread was blocked while closing WAL.
This issue doesn't happen so frequently, but if it happens, it always leads
to huge amount of requests failure. The only way to do is KILL -9.
I think it's a bug, but I haven't found a decent solution. Does anyone have
the same problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-11902) RegionServer was blocked while aborting
[ https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Tian reassigned HBASE-11902: -- Assignee: Qiang Tian RegionServer was blocked while aborting --- Key: HBASE-11902 URL: https://issues.apache.org/jira/browse/HBASE-11902 Project: HBase Issue Type: Bug Components: regionserver, wal Affects Versions: 0.98.4 Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7 Reporter: Victor Xu Assignee: Qiang Tian Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log, jstack_hadoop461.cm6.log Generally, regionserver automatically aborts when isHealth() returns false. But it sometimes got blocked while aborting. I saved the jstack and logs, and found out that it was caused by datanodes failures. The regionserver60020 thread was blocked while closing WAL. This issue doesn't happen so frequently, but if it happens, it always leads to huge amount of requests failure. The only way to do is KILL -9. I think it's a bug, but I haven't found a decent solution. Does anyone have the same problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223982#comment-14223982 ] Hudson commented on HBASE-12550: FAILURE: Integrated in HBase-0.98 #700 (See [https://builds.apache.org/job/HBase-0.98/700/]) HBASE-12550 ADDENDUM Make HRegion's api not change. (elliott: rev 9afe1e8efd25a77c62ca068ed15ad65bb7f06eda) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12404) Task 5 from parent: Replace internal HTable constructor use with HConnection#getTable (0.98, 0.99)
[ https://issues.apache.org/jira/browse/HBASE-12404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12404: -- Attachment: 12404v19.txt Rebase Task 5 from parent: Replace internal HTable constructor use with HConnection#getTable (0.98, 0.99) -- Key: HBASE-12404 URL: https://issues.apache.org/jira/browse/HBASE-12404 Project: HBase Issue Type: Sub-task Reporter: stack Assignee: stack Fix For: 0.99.2 Attachments: 0001-HBASE-12404-Task-5-from-parent-Replace-internal-HTab.patch, 12404.txt, 12404.v16.txt, 12404.v16.txt, 12404.v17.txt, 12404v10.txt, 12404v11.txt, 12404v12.txt, 12404v12.txt, 12404v13.txt, 12404v13.txt, 12404v14.txt, 12404v15.txt, 12404v18.txt, 12404v19.txt, 12404v19.txt, 12404v2.txt, 12404v3.txt, 12404v5.txt, 12404v6.txt, 12404v7.txt, 12404v9.txt Do the step 5. from the [~ndimiduk] list in parent issue. Go through src code and change all new HTable to instead be connection.getTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12550) Check all storefiles are referenced before splitting
[ https://issues.apache.org/jira/browse/HBASE-12550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223988#comment-14223988 ] Hudson commented on HBASE-12550: FAILURE: Integrated in HBase-TRUNK #5814 (See [https://builds.apache.org/job/HBase-TRUNK/5814/]) HBASE-12550 ADDENDUM Make HRegion's api not change. (elliott: rev e83082a88816684714d8a563967046e582f9b8c7) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java Check all storefiles are referenced before splitting Key: HBASE-12550 URL: https://issues.apache.org/jira/browse/HBASE-12550 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 0.98.9, 0.99.2 Attachments: 0001-HBASE-12550-ADDENDUM-Check-all-storefiles-are-refere.patch, 0001-HBASE-12550-ADDENDUM-Make-HRegion-s-api-not-change.patch, HBASE-12550-v1.patch, HBASE-12550-v2.patch, HBASE-12550-v3.patch, HBASE-12550.patch Make double extra sure all storefiles are referenced before -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10881) Support reverse scan in thrift2
[ https://issues.apache.org/jira/browse/HBASE-10881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223987#comment-14223987 ] Ted Yu commented on HBASE-10881: [~apurtell]: Do you want this in 0.98 ? Support reverse scan in thrift2 --- Key: HBASE-10881 URL: https://issues.apache.org/jira/browse/HBASE-10881 Project: HBase Issue Type: New Feature Components: Thrift Affects Versions: 0.99.0 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 0.99.0 Attachments: HBASE-10881-trunk-v1.diff, HBASE-10881-trunk-v2.diff Support reverse scan in thrift2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11902) RegionServer was blocked while aborting
[
https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14223989#comment-14223989
]
Qiang Tian commented on HBASE-11902:
proposed fix for 0.98:
{code}
--- hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1760,7 +1760,13 @@ public class HRegion implements HeapSize { // , Writable{
// sync unflushed WAL changes when deferred log sync is enabled
// see HBASE-8208 for details
if (wal != null !shouldSyncLog()) {
- wal.sync();
+ try {
+wal.sync();
+ } catch (IOException e) {
+ wal.abortCacheFlush(this.getRegionInfo().getEncodedNameAsBytes());
+ LOG.warn(Unexpected exception while wal.sync(), re-throw);
+ throw e;
+ }
}
{code}
the master branch code writes ABORT_FLUSH log before we call
wal.abortCacheFlush. so it is also needed if wal.sync aborts?
also I am thinking about if we could make error injection test for such kind of
failure which could mostly happen in real env but would not happen in UT?
RegionServer was blocked while aborting
---
Key: HBASE-11902
URL: https://issues.apache.org/jira/browse/HBASE-11902
Project: HBase
Issue Type: Bug
Components: regionserver, wal
Affects Versions: 0.98.4
Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7
Reporter: Victor Xu
Assignee: Qiang Tian
Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log,
jstack_hadoop461.cm6.log
Generally, regionserver automatically aborts when isHealth() returns false.
But it sometimes got blocked while aborting. I saved the jstack and logs, and
found out that it was caused by datanodes failures. The regionserver60020
thread was blocked while closing WAL.
This issue doesn't happen so frequently, but if it happens, it always leads
to huge amount of requests failure. The only way to do is KILL -9.
I think it's a bug, but I haven't found a decent solution. Does anyone have
the same problem?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
