[jira] [Commented] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618642#comment-13618642
]
Andrew Purtell commented on HBASE-8213:
---
Green build on 0.94 with {{-Psecurity}}:
http://54.241.6.143/job/HBase-0.94-apurtell/12/
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.7
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8226) HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts too many regions
[ https://issues.apache.org/jira/browse/HBASE-8226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618659#comment-13618659 ] Hudson commented on HBASE-8226: --- Integrated in hbase-0.95 #117 (See [https://builds.apache.org/job/hbase-0.95/117/]) HBASE-8226. HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts 'too many' regions (Revision 1463071) Result = FAILURE apurtell : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorEndpoint.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestMasterObserver.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithAbort.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionServerCoprocessorExceptionWithRemove.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterTransitions.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMiniClusterLoadSequential.java HBaseTestingUtility#waitUntilAllRegionsAssigned won't return if it counts too many regions Key: HBASE-8226 URL: https://issues.apache.org/jira/browse/HBASE-8226 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.95.0, 0.96.0, 0.94.7 Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.95.0, 0.96.0, 0.94.7 Attachments: 8226-branch-0.94.patch, 8226.patch, 8226.patch HBaseTestingUtility#waitUntilAllRegionsAssigned does not check if the region it is counting belongs to the table created by the test and will not return if it accidentally counts too many regions, for example the regions of the ACL table when security is enabled. Made an attempt at fixing this as part of HBASE-8209 but broke TestMasterTransitions. Try again here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618664#comment-13618664
]
Hadoop QA commented on HBASE-8164:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576341/HBASE-8164_addendum_4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.util.TestHBaseFsck.testFixByTable(TestHBaseFsck.java:1159)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5079//console
This message is automatically generated.
TestTableLockManager fails intermittently in trunk builds
-
Key: HBASE-8164
URL: https://issues.apache.org/jira/browse/HBASE-8164
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.95.0, 0.98.0
Attachments: 8164-addendum-2.txt, 8164-addendum.txt, 8164-v2.txt,
8164-v3.txt, 8164-v4.txt, HBASE-8164_addendum_3.patch,
HBASE-8164_addendum_4.patch
In build #3979:
testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test
timed out after 60 milliseconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8104) HBase consistency and availability after replication
[ https://issues.apache.org/jira/browse/HBASE-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618694#comment-13618694 ] Brian Fu commented on HBASE-8104: - [~ctrezzo], one cluster fail refer to hbase and zookeeper cluster are unable, but HDFS cluster is ok. this funciton not like replicaiton . HBase consistency and availability after replication Key: HBASE-8104 URL: https://issues.apache.org/jira/browse/HBASE-8104 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: Brian Fu Priority: Critical Original Estimate: 336h Remaining Estimate: 336h HBase consistency and availability after replication Scene as follows: 1. There are two HBase clusters are the Master clusters and Slave Clusters. two clusters replication function is open. 2. if master cluster have problems, so all write and read request switching to the slave cluster. 3. After a period of time ,we need to switch back to the Master cluster, there will be a part of the data is inconsistent, lead to this part of the data is not available. This feature is particularly important for providing online services HBase cluster. So, I want through a write-back program to keep the data consistency, then to improve HBase availability. we will provide a patch for this function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618693#comment-13618693
]
Jieshan Bean commented on HBASE-8213:
-
[~apurtell]Thank you for the trunk patch. I planned to do that after review:)
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.7
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8220) can we record the count opened HTable for HTablePool
[ https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618699#comment-13618699 ] cuijianwei commented on HBASE-8220: --- Thanks for your comment, I add a new patch which including: commont#1 fixed comment#2 fixed comment#3 add more specific comment at the place we call 'decrementAndGet' comment#4 fixed comment#5 the path is made over 0.94 branch I'm sorry that I don't construct a test suite environment, do we have common platform to run the test suite? can we record the count opened HTable for HTablePool Key: HBASE-8220 URL: https://issues.apache.org/jira/browse/HBASE-8220 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.3 Reporter: cuijianwei Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt In HTablePool, we have a method getCurrentPoolSize(...) to get how many opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable which means the count of HTable get from HTablePool.getTable(...) and don't return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may be meaningful because it indicates how many HTables should be opened for the application which may help us set the appropriate MaxSize of HTablePool. Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8220) can we record the count opened HTable for HTablePool
[ https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cuijianwei updated HBASE-8220: -- Attachment: HBASE-8220-0.94.3.txt-v2 can we record the count opened HTable for HTablePool Key: HBASE-8220 URL: https://issues.apache.org/jira/browse/HBASE-8220 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.3 Reporter: cuijianwei Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt-v2, HBASE-8220-0.94.3-v2.txt In HTablePool, we have a method getCurrentPoolSize(...) to get how many opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable which means the count of HTable get from HTablePool.getTable(...) and don't return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may be meaningful because it indicates how many HTables should be opened for the application which may help us set the appropriate MaxSize of HTablePool. Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8220) can we record the count opened HTable for HTablePool
[ https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cuijianwei updated HBASE-8220: -- Attachment: HBASE-8220-0.94.3-v2.txt rename patch name to *.txt can we record the count opened HTable for HTablePool Key: HBASE-8220 URL: https://issues.apache.org/jira/browse/HBASE-8220 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.3 Reporter: cuijianwei Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt-v2, HBASE-8220-0.94.3-v2.txt In HTablePool, we have a method getCurrentPoolSize(...) to get how many opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable which means the count of HTable get from HTablePool.getTable(...) and don't return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may be meaningful because it indicates how many HTables should be opened for the application which may help us set the appropriate MaxSize of HTablePool. Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8220) can we record the count opened HTable for HTablePool
[ https://issues.apache.org/jira/browse/HBASE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cuijianwei updated HBASE-8220: -- Attachment: HBASE-8220-0.94.3-v3.txt can we record the count opened HTable for HTablePool Key: HBASE-8220 URL: https://issues.apache.org/jira/browse/HBASE-8220 Project: HBase Issue Type: Improvement Components: Client Affects Versions: 0.94.3 Reporter: cuijianwei Attachments: HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt, HBASE-8220-0.94.3.txt-v2, HBASE-8220-0.94.3-v2.txt, HBASE-8220-0.94.3-v3.txt In HTablePool, we have a method getCurrentPoolSize(...) to get how many opened HTable has been pooled. However, we don't know ConcurrentOpenedHTable which means the count of HTable get from HTablePool.getTable(...) and don't return to HTablePool by PooledTable.close(). The ConcurrentOpenedHTable may be meaningful because it indicates how many HTables should be opened for the application which may help us set the appropriate MaxSize of HTablePool. Therefore, we can and a ConcurrentOpenedHTable as a counter in HTablePool. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8235) Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool
ramkrishna.s.vasudevan created HBASE-8235: - Summary: Adding inmemory CF attribute to LoadTest and PerformanceEvaluation tool Key: HBASE-8235 URL: https://issues.apache.org/jira/browse/HBASE-8235 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Adding the inmemory CF attribute to the LoadTestTool and PerfEvaluation tool will make it extensible to test such inmemory scenarios also. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7579) HTableDescriptor equals method fails if results are returned in a different order
[ https://issues.apache.org/jira/browse/HBASE-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7579: --- Attachment: HBASE-7579-v5.patch HBASE-7579-0.94.patch attaching 94 patch with the HTD and HCD tests and the empty family name check in isLegalFamilyName(). v5 is the rebased trunk patch also removing code in constructors that is already in setName(). HTableDescriptor equals method fails if results are returned in a different order - Key: HBASE-7579 URL: https://issues.apache.org/jira/browse/HBASE-7579 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.95.0, 0.94.6 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Priority: Minor Fix For: 0.95.0, 0.94.7 Attachments: HBASE-7579-0.94.patch, HBASE-7579-v1.patch, HBASE-7579-v2.patch, HBASE-7579-v3.patch, HBASE-7579-v4.patch, HBASE-7579-v5.patch HTableDescriptor's compareTo function compares a set of HColumnDescriptors against another set of HColumnDescriptors. It iterates through both, relying on the fact that they will be in the same order. In my testing, I may have seen this issue come up, so I decided to fix it. It's a straightforward fix. I convert the sets into a hashset for O(1) lookups (at least in theory), then I check that all items in the first set are found in the second. Since the sizes are the same, we know that if all elements showed up in the second set, then they must be equal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7579) HTableDescriptor equals method fails if results are returned in a different order
[
https://issues.apache.org/jira/browse/HBASE-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618752#comment-13618752
]
Hadoop QA commented on HBASE-7579:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576365/HBASE-7579-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5080//console
This message is automatically generated.
HTableDescriptor equals method fails if results are returned in a different
order
-
Key: HBASE-7579
URL: https://issues.apache.org/jira/browse/HBASE-7579
Project: HBase
Issue Type: Bug
Components: Admin
Affects Versions: 0.95.0, 0.94.6
Reporter: Aleksandr Shulman
Assignee: Aleksandr Shulman
Priority: Minor
Fix For: 0.95.0, 0.94.7
Attachments: HBASE-7579-0.94.patch, HBASE-7579-v1.patch,
HBASE-7579-v2.patch, HBASE-7579-v3.patch, HBASE-7579-v4.patch,
HBASE-7579-v5.patch
HTableDescriptor's compareTo function compares a set of HColumnDescriptors
against another set of HColumnDescriptors. It iterates through both, relying
on the fact that they will be in the same order.
In my testing, I may have seen this issue come up, so I decided to fix it.
It's a straightforward fix. I convert the sets into a hashset for O(1)
lookups (at least in theory), then I check that all items in the first set
are found in the second.
Since the sizes are the same, we know that if all elements showed up in the
second set, then they must be equal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8174) Allow each table to customize its own flush blocking file count hbase.hstore.blockingStoreFiles
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] clockfly updated HBASE-8174: Attachment: hbase-8174.patch Allow each table to customize its own flush blocking file count hbase.hstore.blockingStoreFiles - Key: HBASE-8174 URL: https://issues.apache.org/jira/browse/HBASE-8174 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: clockfly Assignee: clockfly Priority: Minor Fix For: 0.94.7 Attachments: hbase-8174.patch, HBASE-8174.patch.0.94.v1 Currently, the blocking file count hbase.hstore.blockingStoreFiles is configured at region server level. We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8174) Backport HBASE-8161(setting blocking file count on table level doesn't work)
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] clockfly updated HBASE-8174: Summary: Backport HBASE-8161(setting blocking file count on table level doesn't work) (was: Allow each table to customize its own flush blocking file count hbase.hstore.blockingStoreFiles) Backport HBASE-8161(setting blocking file count on table level doesn't work) Key: HBASE-8174 URL: https://issues.apache.org/jira/browse/HBASE-8174 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: clockfly Assignee: clockfly Priority: Minor Fix For: 0.94.7 Attachments: hbase-8174.patch, HBASE-8174.patch.0.94.v1 Currently, the blocking file count hbase.hstore.blockingStoreFiles is configured at region server level. We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8174) Backport HBASE-8161(setting blocking file count on table level doesn't work)
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618754#comment-13618754 ] clockfly commented on HBASE-8174: - backport patch for 0.94 updated. https://issues.apache.org/jira/secure/attachment/12576375/hbase-8174.patch Backport HBASE-8161(setting blocking file count on table level doesn't work) Key: HBASE-8174 URL: https://issues.apache.org/jira/browse/HBASE-8174 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: clockfly Assignee: clockfly Priority: Minor Fix For: 0.94.7 Attachments: hbase-8174.patch, HBASE-8174.patch.0.94.v1 Currently, the blocking file count hbase.hstore.blockingStoreFiles is configured at region server level. We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7579) HTableDescriptor equals method fails if results are returned in a different order
[ https://issues.apache.org/jira/browse/HBASE-7579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618755#comment-13618755 ] Matteo Bertozzi commented on HBASE-7579: the failures are related to v5... TestAdmin.testTableNames() relies on -ROOT- and .META. to be an invalid name, so you can instantiate them just by using the copy constructor... TestTableLockManager.testTableReadLock() seems to rely on a unstrict HTD check HTableDescriptor equals method fails if results are returned in a different order - Key: HBASE-7579 URL: https://issues.apache.org/jira/browse/HBASE-7579 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.95.0, 0.94.6 Reporter: Aleksandr Shulman Assignee: Aleksandr Shulman Priority: Minor Fix For: 0.95.0, 0.94.7 Attachments: HBASE-7579-0.94.patch, HBASE-7579-v1.patch, HBASE-7579-v2.patch, HBASE-7579-v3.patch, HBASE-7579-v4.patch, HBASE-7579-v5.patch HTableDescriptor's compareTo function compares a set of HColumnDescriptors against another set of HColumnDescriptors. It iterates through both, relying on the fact that they will be in the same order. In my testing, I may have seen this issue come up, so I decided to fix it. It's a straightforward fix. I convert the sets into a hashset for O(1) lookups (at least in theory), then I check that all items in the first set are found in the second. Since the sizes are the same, we know that if all elements showed up in the second set, then they must be equal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7415) [snapshots] Add task information to snapshot operation
[ https://issues.apache.org/jira/browse/HBASE-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7415: --- Attachment: HBASE-7415-v4.patch HBASE-7415-0.94-v1.patch [snapshots] Add task information to snapshot operation -- Key: HBASE-7415 URL: https://issues.apache.org/jira/browse/HBASE-7415 Project: HBase Issue Type: New Feature Components: Client, master, regionserver, snapshots, Zookeeper Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.95.0 Attachments: HBASE-7415-0.94.patch, HBASE-7415-0.94-v1.patch, hbase-7415-v0.patch, hbase-7415-v1.patch, HBASE-7415-v1-rebase.patch, HBASE-7415-v2.patch, HBASE-7415-v3.patch, HBASE-7415-v4.patch, HBase_Snapshot_Task_UI.png Snapshot operations should have some sort of progresss information available via the WebUI so admins can track progress of operations. This should go a long way to enable 'good' admins to not hose their clusters by running concurrent snapshot operations (e.g. rename while a clone is in progress). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7615) Add metrics for snapshots
[ https://issues.apache.org/jira/browse/HBASE-7615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7615: --- Attachment: HBASE-7615-v3.patch HBASE-7615-0.94-v1.patch Add metrics for snapshots - Key: HBASE-7615 URL: https://issues.apache.org/jira/browse/HBASE-7615 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Assignee: Matteo Bertozzi Attachments: HBASE-7615-0.94.patch, HBASE-7615-0.94-v1.patch, HBASE-7615-v0.patch, HBASE-7615-v0-ui-corrupted.png, HBASE-7615-v0-ui.png, HBASE-7615-v1.patch, HBASE-7615-v2.patch, HBASE-7615-v3.patch Metrics should be added for snapshot. From Matteo: Output that we have in SnapshotInfo should be covered: %d HFiles (%d in archive), total size %s (%.2f%% %s shared with the source table) Jesse mentioned snaphot counts, average time to completion. I think we should have counts for successful / failed snapshots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[
https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618798#comment-13618798
]
Ted Yu commented on HBASE-8164:
---
Addendum 4 looks good.
{code}
+ region got closed and the attempts got over before
++ the region could have got reassigned.);
{code}
There should be a space between before and the.
{code}
+ if(regCount 0){
{code}
nit: space between 'if' and '('
TestTableLockManager fails intermittently in trunk builds
-
Key: HBASE-8164
URL: https://issues.apache.org/jira/browse/HBASE-8164
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.95.0, 0.98.0
Attachments: 8164-addendum-2.txt, 8164-addendum.txt, 8164-v2.txt,
8164-v3.txt, 8164-v4.txt, HBASE-8164_addendum_3.patch,
HBASE-8164_addendum_4.patch
In build #3979:
testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test
timed out after 60 milliseconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7925) Back port HBASE-6881 into 0.94
[ https://issues.apache.org/jira/browse/HBASE-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618810#comment-13618810 ] Ted Yu commented on HBASE-7925: --- Integrated to 0.94 Thanks for the patch, Rajeshbabu. Back port HBASE-6881 into 0.94 -- Key: HBASE-7925 URL: https://issues.apache.org/jira/browse/HBASE-7925 Project: HBase Issue Type: Bug Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.94.7 Attachments: HBASE-7925_94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8219) Align Offline Merge with Online Merge
[ https://issues.apache.org/jira/browse/HBASE-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618867#comment-13618867 ] Ted Yu commented on HBASE-8219: --- +1 from me. Align Offline Merge with Online Merge - Key: HBASE-8219 URL: https://issues.apache.org/jira/browse/HBASE-8219 Project: HBase Issue Type: Task Components: regionserver Affects Versions: 0.95.0 Reporter: Matteo Bertozzi Assignee: chunhui shen Attachments: hbase-8219v1.patch, hbase-8219v2.patch After HBASE-7403 we now have two different tools for online and offline merge, and the result produced by the two are different. (the online one works with snapshots, the offline not) We should remove the offline one, or align it to the online code. Most of the offline code in HRegion.merge() can be replaced with the one in RegionMergeTransaction, used by the online version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618878#comment-13618878
]
Andrew Purtell commented on HBASE-8213:
---
Hope you don't mind [~jeason], wanted to get a HadoopQA run in.
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.7
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-8213:
--
Affects Version/s: (was: 0.94.7)
0.94.6
Fix Version/s: 0.94.7
0.98.0
0.95.0
Hadoop Flags: Reviewed
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Fix For: 0.95.0, 0.98.0, 0.94.7
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618882#comment-13618882
]
Ted Yu commented on HBASE-8213:
---
Patch is good.
{code}
+final HBaseAdmin admin = TEST_UTIL.getHBaseAdmin();
{code}
admin is not closed in the test.
{code}
+ public void testGlobalAuthorizationForNewRegisteredRS() throws Exception {
{code}
nit: I think the method should be named
testGlobalAuthorizationForNewlyRegisteredRS
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Fix For: 0.95.0, 0.98.0, 0.94.7
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8164: - Attachment: HBASE-8164_addendum_5.patch What I applied. Includes fixes for Ted nits. TestTableLockManager fails intermittently in trunk builds - Key: HBASE-8164 URL: https://issues.apache.org/jira/browse/HBASE-8164 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.95.0, 0.98.0 Attachments: 8164-addendum-2.txt, 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt, HBASE-8164_addendum_3.patch, HBASE-8164_addendum_4.patch, HBASE-8164_addendum_5.patch In build #3979: testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7896) make rename_table working in 92/94
[ https://issues.apache.org/jira/browse/HBASE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1361#comment-1361 ] Tianying Chang commented on HBASE-7896: --- I feel no need for the ruby script if we have the function exposed through HBase shell. make rename_table working in 92/94 -- Key: HBASE-7896 URL: https://issues.apache.org/jira/browse/HBASE-7896 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.2, 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.92.3, 0.94.7 Attachments: rename_table.rb The rename_table function is very useful for our customers. However, rename_table.rb does not work for 92/94. It has several bugs. It will be useful to fix them so that users can solve their problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8164: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for addendum [~ram_krish] Blessed be the unit test fixers! TestTableLockManager fails intermittently in trunk builds - Key: HBASE-8164 URL: https://issues.apache.org/jira/browse/HBASE-8164 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.95.0, 0.98.0 Attachments: 8164-addendum-2.txt, 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt, HBASE-8164_addendum_3.patch, HBASE-8164_addendum_4.patch, HBASE-8164_addendum_5.patch In build #3979: testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7896) make rename_table working in 92/94
[ https://issues.apache.org/jira/browse/HBASE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618890#comment-13618890 ] Tianying Chang commented on HBASE-7896: --- I am almost done for the HBA version, with one unit test added with it. I will post the patch in couple days. make rename_table working in 92/94 -- Key: HBASE-7896 URL: https://issues.apache.org/jira/browse/HBASE-7896 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.2, 0.94.5 Reporter: Tianying Chang Assignee: Tianying Chang Fix For: 0.92.3, 0.94.7 Attachments: rename_table.rb The rename_table function is very useful for our customers. However, rename_table.rb does not work for 92/94. It has several bugs. It will be useful to fix them so that users can solve their problems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-8213:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Fix For: 0.95.0, 0.98.0, 0.94.7
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Attachment: HBASE-8127_94_3.patch
Patch Addressing Jeffrey's comments. Please review.
Region of a disabling or disabled table could be stuck in transition state
when RS dies during Master initialization
Key: HBASE-8127
URL: https://issues.apache.org/jira/browse/HBASE-8127
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Jeffrey Zhong
Assignee: rajeshbabu
Fix For: 0.94.7
Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
The issue happens when a RS dies during a master starts up. After the RS
reports open to the new master instance and dies immediately thereafter, the
RITs of disabling tables(or disabled table) on the died RS will be in RIT
state forever.
I attached a patch to simulate the situation and you can run the following
command to reproduce the issue:
{code}mvn test -PlocalTests
-Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
Basically, we skip regions of a dead server inside
AM.processDeadServersAndRecoverLostRegions as the following code and relies
on SSH to process those skipped regions:
{code}
for (PairHRegionInfo, Result deadRegion : deadServer.getValue()) {
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
While in SSH, we skip regions of disabling(or disabled table) again by
function processDeadRegion. Finally comes to the issue that RITs of
disabling(or disabled table) stuck there forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-1936) ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services
[ https://issues.apache.org/jira/browse/HBASE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HBASE-1936: -- Assignee: Jimmy Xiang ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services --- Key: HBASE-1936 URL: https://issues.apache.org/jira/browse/HBASE-1936 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Jimmy Xiang Labels: noob Attachments: cp_from_hdfs.patch, HBASE-1936-trunk(forReview).patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7896) make rename_table working in 92/94
[
https://issues.apache.org/jira/browse/HBASE-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618902#comment-13618902
]
Matteo Bertozzi commented on HBASE-7896:
as I've said in the beginning, if we implement the rename as the current rb
script...
a rename will cause a data loss for snapshots and clones on the renamed table.
A simple implementation of rename in 0.94 is:
{code}
rename(String oldTableName, String newTableName) {
String snapshotName = randomName();
snapshot(snapshotName, oldTableName);
cloneSnapshot(snapshotName, newTableName);
deleteSnapshot(snapshotName);
deleteTable(oldTableName)
}
{code}
make rename_table working in 92/94
--
Key: HBASE-7896
URL: https://issues.apache.org/jira/browse/HBASE-7896
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.92.2, 0.94.5
Reporter: Tianying Chang
Assignee: Tianying Chang
Fix For: 0.92.3, 0.94.7
Attachments: rename_table.rb
The rename_table function is very useful for our customers. However,
rename_table.rb does not work for 92/94. It has several bugs. It will be
useful to fix them so that users can solve their problems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618903#comment-13618903
]
ramkrishna.s.vasudevan commented on HBASE-8127:
---
I will review this today or my time tomorrow morning. Also if things are fine
will commit the patch.
Jeffrey, if you have comments on the latest feel free to comment on it.
Region of a disabling or disabled table could be stuck in transition state
when RS dies during Master initialization
Key: HBASE-8127
URL: https://issues.apache.org/jira/browse/HBASE-8127
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Jeffrey Zhong
Assignee: rajeshbabu
Fix For: 0.94.7
Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
The issue happens when a RS dies during a master starts up. After the RS
reports open to the new master instance and dies immediately thereafter, the
RITs of disabling tables(or disabled table) on the died RS will be in RIT
state forever.
I attached a patch to simulate the situation and you can run the following
command to reproduce the issue:
{code}mvn test -PlocalTests
-Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
Basically, we skip regions of a dead server inside
AM.processDeadServersAndRecoverLostRegions as the following code and relies
on SSH to process those skipped regions:
{code}
for (PairHRegionInfo, Result deadRegion : deadServer.getValue()) {
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
While in SSH, we skip regions of disabling(or disabled table) again by
function processDeadRegion. Finally comes to the issue that RITs of
disabling(or disabled table) stuck there forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1936) ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services
[ https://issues.apache.org/jira/browse/HBASE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618907#comment-13618907 ] Jimmy Xiang commented on HBASE-1936: I agree. I will take a look. I was thinking to reuse some logic in the coprocessor framework. The new class loader (most likely the existing one in coprocessor framework, with some refactory), will be used to load some non-exempt classes (not system classes) from a configurable location, which could be a local folder, or a HDFS folder. Ideally, this loader will be used only for get/scan calls in region server side (besides coprocessor related). ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services --- Key: HBASE-1936 URL: https://issues.apache.org/jira/browse/HBASE-1936 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Jimmy Xiang Labels: noob Attachments: cp_from_hdfs.patch, HBASE-1936-trunk(forReview).patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8174) Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8174: -- Summary: Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94 (was: Backport HBASE-8161(setting blocking file count on table level doesn't work)) Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94 Key: HBASE-8174 URL: https://issues.apache.org/jira/browse/HBASE-8174 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: clockfly Assignee: clockfly Priority: Minor Fix For: 0.94.7 Attachments: hbase-8174.patch, HBASE-8174.patch.0.94.v1 Currently, the blocking file count hbase.hstore.blockingStoreFiles is configured at region server level. We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8174) Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618917#comment-13618917 ] Ted Yu commented on HBASE-8174: --- Integrated to 0.94 Thanks for the patch, clockfly. Thanks for the reviews, Sergey, Lars and Anoop. Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94 Key: HBASE-8174 URL: https://issues.apache.org/jira/browse/HBASE-8174 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: clockfly Assignee: clockfly Priority: Minor Fix For: 0.94.7 Attachments: hbase-8174.patch, HBASE-8174.patch.0.94.v1 Currently, the blocking file count hbase.hstore.blockingStoreFiles is configured at region server level. We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8217) Port online region merge (HBASE-7403) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618923#comment-13618923 ] Andrew Purtell commented on HBASE-8217: --- Late to the discussion, sorry. +1 (or rather -1) to closing this wontfix on account of the risk assessment. -1 on the notion of anyone unilaterally declaring 0.94 feature complete Port online region merge (HBASE-7403) to 0.94 - Key: HBASE-8217 URL: https://issues.apache.org/jira/browse/HBASE-8217 Project: HBase Issue Type: New Feature Components: master, Region Assignment, regionserver Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: hbase-8217_v2.patch HBASE-7403 added online region merge, and there was some discussion about whether we can port this to 0.94. In this issue, we can discuss feasibility and decide one what way or the other. I actually have a patch on top of backported HBASE-7721. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8133) avoid assign for disabling table regions(OPENING/PENDING_OPEN) in SSH
[ https://issues.apache.org/jira/browse/HBASE-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-8133: -- Attachment: HBASE-8133_2.patch Rebased patch. [~jxiang],[~ram_krish] Can you give your opinion about this patch? avoid assign for disabling table regions(OPENING/PENDING_OPEN) in SSH - Key: HBASE-8133 URL: https://issues.apache.org/jira/browse/HBASE-8133 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 0.95.0 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0 Attachments: HBASE-8133_2.patch, HBASE-8133.patch Disabling table regions which are in PENDING_OPEN or OPENING on dead server are getting assigned then unassiging. We can avoid this by just mark OFFLINE for the regions,any way znodes of the transitions got deleted as part of am.processServerShutdown(serverName). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7835) Implementation of the log splitting without creating intermediate files
[ https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618928#comment-13618928 ] Ted Yu commented on HBASE-7835: --- Latest patch passed test suite. +1 Implementation of the log splitting without creating intermediate files Key: HBASE-7835 URL: https://issues.apache.org/jira/browse/HBASE-7835 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.95.0 Attachments: hbase-7006_12.patch, hbase-7006_13.patch, hbase-7006_1.patch, hbase-7006_2.patch, hbase-7006_3.patch, hbase-7006_3.patch, hbase-7006_6.patch, hbase-7006_8.patch The sub task is to check in a separate patch for major implementation for the proposal. More checkins will include a new replay command and more metrics support. Thanks, -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618932#comment-13618932
]
Ted Yu commented on HBASE-8127:
---
I got the following based on latest patch:
{code}
testMasterFailoverWhenDisablingTableRegionsInRITOnDeadRS(org.apache.hadoop.hbase.regionserver.TestRSKilledWhenMasterInitializing):
test timed out after 18 milliseconds
{code}
Region of a disabling or disabled table could be stuck in transition state
when RS dies during Master initialization
Key: HBASE-8127
URL: https://issues.apache.org/jira/browse/HBASE-8127
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Jeffrey Zhong
Assignee: rajeshbabu
Fix For: 0.94.7
Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
The issue happens when a RS dies during a master starts up. After the RS
reports open to the new master instance and dies immediately thereafter, the
RITs of disabling tables(or disabled table) on the died RS will be in RIT
state forever.
I attached a patch to simulate the situation and you can run the following
command to reproduce the issue:
{code}mvn test -PlocalTests
-Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
Basically, we skip regions of a dead server inside
AM.processDeadServersAndRecoverLostRegions as the following code and relies
on SSH to process those skipped regions:
{code}
for (PairHRegionInfo, Result deadRegion : deadServer.getValue()) {
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
While in SSH, we skip regions of disabling(or disabled table) again by
function processDeadRegion. Finally comes to the issue that RITs of
disabling(or disabled table) stuck there forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7835) Implementation of the log splitting without creating intermediate files
[ https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618939#comment-13618939 ] Jeffrey Zhong commented on HBASE-7835: -- Thanks [~te...@apache.org]! [~saint@gmail.com] Now the ball is in your court. Thanks in advance! Implementation of the log splitting without creating intermediate files Key: HBASE-7835 URL: https://issues.apache.org/jira/browse/HBASE-7835 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jeffrey Zhong Assignee: Jeffrey Zhong Fix For: 0.95.0 Attachments: hbase-7006_12.patch, hbase-7006_13.patch, hbase-7006_1.patch, hbase-7006_2.patch, hbase-7006_3.patch, hbase-7006_3.patch, hbase-7006_6.patch, hbase-7006_8.patch The sub task is to check in a separate patch for major implementation for the proposal. More checkins will include a new replay command and more metrics support. Thanks, -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8133) avoid assign for disabling table regions(OPENING/PENDING_OPEN) in SSH
[ https://issues.apache.org/jira/browse/HBASE-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618942#comment-13618942 ] Jimmy Xiang commented on HBASE-8133: +1 avoid assign for disabling table regions(OPENING/PENDING_OPEN) in SSH - Key: HBASE-8133 URL: https://issues.apache.org/jira/browse/HBASE-8133 Project: HBase Issue Type: Bug Components: Region Assignment Affects Versions: 0.95.0 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.98.0 Attachments: HBASE-8133_2.patch, HBASE-8133.patch Disabling table regions which are in PENDING_OPEN or OPENING on dead server are getting assigned then unassiging. We can avoid this by just mark OFFLINE for the regions,any way znodes of the transitions got deleted as part of am.processServerShutdown(serverName). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
[ https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618946#comment-13618946 ] Chris Trezzo commented on HBASE-8229: - bq. For this issue, I'll just add the same waiting we do when the peer is down (which is the same logical behavior we currently have, but without the insane busy retrying). +1 Replication code logs like crazy if a target table cannot be found. --- Key: HBASE-8229 URL: https://issues.apache.org/jira/browse/HBASE-8229 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 One of our RS/DN machines ran out of diskspace on the partition to which we write the log files. It turns out we still had a table in our source cluster with REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster. In then logged a long stack trace every 50ms or so, over a few days that filled up our log partition. Since ReplicationSource cannot make any progress in this case anyway, it should probably sleep a bit before retrying (or at least limit the rate at which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
[ https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618948#comment-13618948 ] Himanshu Vashishtha commented on HBASE-8229: Yea, I wonder if we show a replication tab on master UI or somewhere which shows some kind of replication state. Basic errors like table not existing on slave, missing family, or may be exception thrown by the slave at peer level are shown. That should give some basic idea to the user. What others think about this? Replication code logs like crazy if a target table cannot be found. --- Key: HBASE-8229 URL: https://issues.apache.org/jira/browse/HBASE-8229 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 One of our RS/DN machines ran out of diskspace on the partition to which we write the log files. It turns out we still had a table in our source cluster with REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster. In then logged a long stack trace every 50ms or so, over a few days that filled up our log partition. Since ReplicationSource cannot make any progress in this case anyway, it should probably sleep a bit before retrying (or at least limit the rate at which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8174) Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618953#comment-13618953 ] Sergey Shelukhin commented on HBASE-8174: - um, sorry to respond late, by still may not work above I meant that both will probably be necessary. HBASE-8161 requires ability to add settings on table or CF, which 0.94 doesn't have (correct me if I'm wrong). HBASE-5335 adds one of them and HBASE-7236 sub-jiras add both but much more intrusively. Backport HBASE-8161(setting blocking file count on table level doesn't work) to 0.94 Key: HBASE-8174 URL: https://issues.apache.org/jira/browse/HBASE-8174 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: clockfly Assignee: clockfly Priority: Minor Fix For: 0.94.7 Attachments: hbase-8174.patch, HBASE-8174.patch.0.94.v1 Currently, the blocking file count hbase.hstore.blockingStoreFiles is configured at region server level. We should allow it to be configured at Table level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
[ https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618956#comment-13618956 ] Chris Trezzo commented on HBASE-8229: - I like the idea of a basic admin tab a lot. Also, maybe the list of peer clusters being replicated to, the ability to dump a list of hlogs in a queue, etc. etc. Replication code logs like crazy if a target table cannot be found. --- Key: HBASE-8229 URL: https://issues.apache.org/jira/browse/HBASE-8229 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 One of our RS/DN machines ran out of diskspace on the partition to which we write the log files. It turns out we still had a table in our source cluster with REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster. In then logged a long stack trace every 50ms or so, over a few days that filled up our log partition. Since ReplicationSource cannot make any progress in this case anyway, it should probably sleep a bit before retrying (or at least limit the rate at which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618960#comment-13618960
]
rajeshbabu commented on HBASE-8127:
---
@Ted,
I ran multiple times before uploading patch then its passed. But Now when I ran
it,got the same problem.
Thanks.
Here the problem is SSH started after master initialization. Same problem
exists with latest patch.
{code}
+if ((rit != null || !((HMaster) this.server).isInitialized()) disabled)
{
{code}
I think
https://issues.apache.org/jira/secure/attachment/12575180/HBASE-8127_94_2.patch
will be better.
[~jeffreyz]
what do you say?
Region of a disabling or disabled table could be stuck in transition state
when RS dies during Master initialization
Key: HBASE-8127
URL: https://issues.apache.org/jira/browse/HBASE-8127
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Jeffrey Zhong
Assignee: rajeshbabu
Fix For: 0.94.7
Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
The issue happens when a RS dies during a master starts up. After the RS
reports open to the new master instance and dies immediately thereafter, the
RITs of disabling tables(or disabled table) on the died RS will be in RIT
state forever.
I attached a patch to simulate the situation and you can run the following
command to reproduce the issue:
{code}mvn test -PlocalTests
-Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
Basically, we skip regions of a dead server inside
AM.processDeadServersAndRecoverLostRegions as the following code and relies
on SSH to process those skipped regions:
{code}
for (PairHRegionInfo, Result deadRegion : deadServer.getValue()) {
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
While in SSH, we skip regions of disabling(or disabled table) again by
function processDeadRegion. Finally comes to the issue that RITs of
disabling(or disabled table) stuck there forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8104) HBase consistency and availability after replication
[ https://issues.apache.org/jira/browse/HBASE-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618963#comment-13618963 ] Chris Trezzo commented on HBASE-8104: - Ah I see. HBase consistency and availability after replication Key: HBASE-8104 URL: https://issues.apache.org/jira/browse/HBASE-8104 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: Brian Fu Priority: Critical Original Estimate: 336h Remaining Estimate: 336h HBase consistency and availability after replication Scene as follows: 1. There are two HBase clusters are the Master clusters and Slave Clusters. two clusters replication function is open. 2. if master cluster have problems, so all write and read request switching to the slave cluster. 3. After a period of time ,we need to switch back to the Master cluster, there will be a part of the data is inconsistent, lead to this part of the data is not available. This feature is particularly important for providing online services HBase cluster. So, I want through a write-back program to keep the data consistency, then to improve HBase availability. we will provide a patch for this function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
[ https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618964#comment-13618964 ] Himanshu Vashishtha commented on HBASE-8229: Hey Chris: Thanks for response :) I'll create a jira for that and add it to the description. For a dump of list of hlogs, we do have it in zkdump link on the master UI (HBASE-7540). Replication code logs like crazy if a target table cannot be found. --- Key: HBASE-8229 URL: https://issues.apache.org/jira/browse/HBASE-8229 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 One of our RS/DN machines ran out of diskspace on the partition to which we write the log files. It turns out we still had a table in our source cluster with REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster. In then logged a long stack trace every 50ms or so, over a few days that filled up our log partition. Since ReplicationSource cannot make any progress in this case anyway, it should probably sleep a bit before retrying (or at least limit the rate at which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7667) Support stripe compaction
[ https://issues.apache.org/jira/browse/HBASE-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618967#comment-13618967 ] Sergey Shelukhin commented on HBASE-7667: - Currently only one compaction per store is allowed. The need to compact several stripes in parallel can probably be alleviated by just having less stripes? As future improvement it is possible to add. Support stripe compaction - Key: HBASE-7667 URL: https://issues.apache.org/jira/browse/HBASE-7667 Project: HBase Issue Type: New Feature Components: Compaction Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: Stripe compaction perf evaluation.pdf, Stripe compaction perf evaluation.pdf, Stripe compactions.pdf, Stripe compactions.pdf, Stripe compactions.pdf So I was thinking about having many regions as the way to make compactions more manageable, and writing the level db doc about how level db range overlap and data mixing breaks seqNum sorting, and discussing it with Jimmy, Matteo and Ted, and thinking about how to avoid Level DB I/O multiplication factor. And I suggest the following idea, let's call it stripe compactions. It's a mix between level db ideas and having many small regions. It allows us to have a subset of benefits of many regions (wrt reads and compactions) without many of the drawbacks (managing and current memstore/etc. limitation). It also doesn't break seqNum-based file sorting for any one key. It works like this. The region key space is separated into configurable number of fixed-boundary stripes (determined the first time we stripe the data, see below). All the data from memstores is written to normal files with all keys present (not striped), similar to L0 in LevelDb, or current files. Compaction policy does 3 types of compactions. First is L0 compaction, which takes all L0 files and breaks them down by stripe. It may be optimized by adding more small files from different stripes, but the main logical outcome is that there are no more L0 files and all data is striped. Second is exactly similar to current compaction, but compacting one single stripe. In future, nothing prevents us from applying compaction rules and compacting part of the stripe (e.g. similar to current policy with rations and stuff, tiers, whatever), but for the first cut I'd argue let it major compact the entire stripe. Or just have the ratio and no more complexity. Finally, the third addresses the concern of the fixed boundaries causing stripes to be very unbalanced. It's exactly like the 2nd, except it takes 2+ adjacent stripes and writes the results out with different boundaries. There's a tradeoff here - if we always take 2 adjacent stripes, compactions will be smaller but rebalancing will take ridiculous amount of I/O. If we take many stripes we are essentially getting into the epic-major-compaction problem again. Some heuristics will have to be in place. In general, if, before stripes are determined, we initially let L0 grow before determining the stripes, we will get better boundaries. Also, unless unbalancing is really large we don't need to rebalance really. Obviously this scheme (as well as level) is not applicable for all scenarios, e.g. if timestamp is your key it completely falls apart. The end result: - many small compactions that can be spread out in time. - reads still read from a small number of files (one stripe + L0). - region splits become marvelously simple (if we could move files between regions, no references would be needed). Main advantage over Level (for HBase) is that default store can still open the files and get correct results - there are no range overlap shenanigans. It also needs no metadata, although we may record some for convenience. It also would appear to not cause as much I/O. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8229) Replication code logs like crazy if a target table cannot be found.
[ https://issues.apache.org/jira/browse/HBASE-8229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618970#comment-13618970 ] Chris Trezzo commented on HBASE-8229: - Good point, I guess that would be sort of redundant. Although, I was thinking of something that isn't tied to ZK (i.e. a way to view the outstanding hlogs that need to be replicated without having to understand the structure of the ZK nodes or how replication queues are stored). Replication code logs like crazy if a target table cannot be found. --- Key: HBASE-8229 URL: https://issues.apache.org/jira/browse/HBASE-8229 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.95.0, 0.98.0, 0.94.7 One of our RS/DN machines ran out of diskspace on the partition to which we write the log files. It turns out we still had a table in our source cluster with REPLICATION_SCOPE=1 that did not have a matching table in the remote cluster. In then logged a long stack trace every 50ms or so, over a few days that filled up our log partition. Since ReplicationSource cannot make any progress in this case anyway, it should probably sleep a bit before retrying (or at least limit the rate at which it spews out these exceptions to the log). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618975#comment-13618975
]
rajeshbabu commented on HBASE-6469:
---
[~jxiang]
bq. I think it is because the ZKTable states are cached.
Here even zktable states are cached still they are same as znode states in zk.
bq. we can manually reset the table state in ZK, then retry will be able to
move on.
Can you give more details how we can do this? How HBASE-8233 really solves this
problem?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[
https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618979#comment-13618979
]
stack commented on HBASE-8224:
--
Parent version must be hard-coded throughout
(http://jira.codehaus.org/browse/MNG-624).
The versions plugin rewrites the pom version in parent and in submodules for
you (have to use 1.3.1, the 2.0 is broke NPE'ing --
https://jira.codehaus.org/browse/MVERSIONS-201)
{code}
$ mvn clean org.codehaus.mojo:versions-maven-plugin:1.3.1:set
-DnewVersion=1.2.3-hadoop2-SNAPSHOT -Dhadoop.profile=2.0 install -DskipTests
assembly:single
{code}
So, unless someone has a better idea, I'm thinking that when we build to
publish, we use this ugly versions plugin to create jars w/ -hadoop1 suffix and
-hadoop2 suffix. I suppose will have to commit what the versions plugin writes
and then tag. Do it once for hadoop1 and once for hadoop2.
Let me try this for first 0.95RC
Add '-hadoop1' or '-hadoop2' to our version string
--
Key: HBASE-8224
URL: https://issues.apache.org/jira/browse/HBASE-8224
Project: HBase
Issue Type: Task
Reporter: stack
Attachments: 8224-adding.classifiers.txt
So we can publish both the hadoop1 and the hadoop2 jars to a maven
repository, and so we can publish two packages, one for hadoop1 and one for
hadoop2, given how maven works, our only alternative (to the best of my
knowledge and after consulting others) is by amending the version string to
include hadoop1 or hadoop2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618980#comment-13618980
]
Jimmy Xiang commented on HBASE-6469:
[~rajesh23], if we manually change the znode state in zk from zkcli, will the
zktable state be updated? Do we have a watcher on that znode? If so, we can
manually set the table to be disabled/enabled, then from hbase shell to retry
enable/disable the table. Will this work?
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[ https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618982#comment-13618982 ] Enis Soztutar commented on HBASE-8224: -- Sounds good to me. Can we automagically invoke this thing when the hadoop2 profile is on? Add '-hadoop1' or '-hadoop2' to our version string -- Key: HBASE-8224 URL: https://issues.apache.org/jira/browse/HBASE-8224 Project: HBase Issue Type: Task Reporter: stack Attachments: 8224-adding.classifiers.txt So we can publish both the hadoop1 and the hadoop2 jars to a maven repository, and so we can publish two packages, one for hadoop1 and one for hadoop2, given how maven works, our only alternative (to the best of my knowledge and after consulting others) is by amending the version string to include hadoop1 or hadoop2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8233) Poke the cache and let the system reload cached information
[ https://issues.apache.org/jira/browse/HBASE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618986#comment-13618986 ] Nick Dimiduk commented on HBASE-8233: - This sounds like an anti-fix to me. Instead, change updates should be propagated correctly. Isn't this kind of thing what ZK watches are designed to support? Poke the cache and let the system reload cached information --- Key: HBASE-8233 URL: https://issues.apache.org/jira/browse/HBASE-8233 Project: HBase Issue Type: New Feature Reporter: Jimmy Xiang Sometimes, cached information is stale and we don't have a way to get rid of it except restarting the server. If we can poke the cache and let the system reloads, it will much easier and quicker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8192) wrong logic in HRegion.bulkLoadHFiles(List)
[
https://issues.apache.org/jira/browse/HBASE-8192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13618987#comment-13618987
]
Chenghao Jiang commented on HBASE-8192:
---
So what should I do next?
wrong logic in HRegion.bulkLoadHFiles(List)
---
Key: HBASE-8192
URL: https://issues.apache.org/jira/browse/HBASE-8192
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.1, 0.94.2, 0.94.3, 0.94.4, 0.94.5
Reporter: Chenghao Jiang
Assignee: Chenghao Jiang
Labels: patch
Fix For: 0.94.7
Attachments: 8192.txt, 8192-v2-with-a-test-case.txt,
8192-v3-with-a-test-case.txt, 8192-v4-with-a-test-case.txt,
8192-v5-with-a-test-case.txt, 8192-v6-with-a-test-case.txt,
8192-v7-with-a-test-case.txt, 8192-v8-with-a-test-case.txt
the wrong logic is here:
when a ColumnFamily does not exist, it gets a null store object, then
ioes.add(ioe); failures.add(p)
but the code below, if (failures.size() != 0), it prints a warn log and
return false, so it will never go into the code if (ioes.size() != 0) below,
and IOException will not be thrown, then the client will keep retry forever.
there is the same situation when doing store.assertBulkLoadHFileOk, if any
WrongRegionException is caught and failures.add(p), then all the other
IOException thrown by assertBulkLoadHFileOk will be ignored.
so i think if (failures.size() != 0) {} should be dealt with after if
(ioes.size() !=0) {}
{code}
for (Pairbyte[], String p : familyPaths) {
byte[] familyName = p.getFirst();
String path = p.getSecond();
Store store = getStore(familyName);
if (store == null) {
IOException ioe = new DoNotRetryIOException(
No such column family + Bytes.toStringBinary(familyName));
ioes.add(ioe);
failures.add(p);
} else {
try {
store.assertBulkLoadHFileOk(new Path(path));
} catch (WrongRegionException wre) {
// recoverable (file doesn't fit in region)
failures.add(p);
} catch (IOException ioe) {
// unrecoverable (hdfs problem)
ioes.add(ioe);
}
}
}
// validation failed, bail out before doing anything permanent.
if (failures.size() != 0) {
StringBuilder list = new StringBuilder();
for (Pairbyte[], String p : failures) {
list.append(\n).append(Bytes.toString(p.getFirst())).append( : )
.append(p.getSecond());
}
// problem when validating
LOG.warn(There was a recoverable bulk load failure likely due to a +
split. These (family, HFile) pairs were not loaded: + list);
return false;
}
// validation failed because of some sort of IO problem.
if (ioes.size() != 0) {
LOG.error(There were IO errors when checking if bulk load is ok. +
throwing exception!);
throw MultipleIOException.createIOException(ioes);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5583) Master restart on create table with splitkeys does not recreate table with all the splitkey regions
[ https://issues.apache.org/jira/browse/HBASE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619002#comment-13619002 ] ramkrishna.s.vasudevan commented on HBASE-5583: --- I am facing some problems in making the testcases run. Or else would have uploaded a patch. I am still figuring out the root cause. Will keep updated. Master restart on create table with splitkeys does not recreate table with all the splitkey regions --- Key: HBASE-5583 URL: https://issues.apache.org/jira/browse/HBASE-5583 Project: HBase Issue Type: Bug Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.95.0 Attachments: HBASE-5583_new_1.patch, HBASE-5583_new_2.patch, HBASE-5583_new_4_WIP.patch, HBASE-5583_new_5_WIP_using_tableznode.patch - Create table using splitkeys - MAster goes down before all regions are added to meta - On master restart the table is again enabled but with less number of regions than specified in splitkeys Anyway client will get an exception if i had called sync create table. But table exists or not check will say table exists. Is this scenario to be handled by client only or can we have some mechanism on the master side for this? Pls suggest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7572) move metadata settings that duplicate xml config settings to CF/table config in a backward-compatible manner
[ https://issues.apache.org/jira/browse/HBASE-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7572: Attachment: HBASE-7572-v3.patch move metadata settings that duplicate xml config settings to CF/table config in a backward-compatible manner Key: HBASE-7572 URL: https://issues.apache.org/jira/browse/HBASE-7572 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7572-v0.patch, HBASE-7572-v1.patch, HBASE-7572-v2.patch, HBASE-7572-v3.patch 2nd part of splitting HBASE-7236 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7828) Expose HBase Scan object's batch property for intra-row batching in Thrift API
[ https://issues.apache.org/jira/browse/HBASE-7828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivendra Pratap Singh updated HBASE-7828: -- Attachment: hbase_7828_trunk.patch.2 Patch with correct numbering. Expose HBase Scan object's batch property for intra-row batching in Thrift API Key: HBASE-7828 URL: https://issues.apache.org/jira/browse/HBASE-7828 Project: HBase Issue Type: New Feature Components: Thrift Affects Versions: 0.94.0 Reporter: Shivendra Pratap Singh Priority: Minor Labels: Hbase, Thrift Fix For: 0.95.0 Attachments: hbase_7828.patch, hbase_7828_trunk.patch, hbase_7828_trunk.patch.2 Hbase scan object has a property called batch. This property allows intra row batching. This property is not exposed in the Thrift TScan specification. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8233) Poke the cache and let the system reload cached information
[ https://issues.apache.org/jira/browse/HBASE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619008#comment-13619008 ] Jimmy Xiang commented on HBASE-8233: ZK watcher is an alternative. But we don't want too many ZK watchers, which may have performance impact. For the ZKTable state specific, the cache should match the znode all the time since the updates are expected to from the code, not from zkcli. So generally, a ZK watcher is not needed. If we have a watcher just for this scenario (HBASE-6469), it may be a waste. Poke the cache and let the system reload cached information --- Key: HBASE-8233 URL: https://issues.apache.org/jira/browse/HBASE-8233 Project: HBase Issue Type: New Feature Reporter: Jimmy Xiang Sometimes, cached information is stale and we don't have a way to get rid of it except restarting the server. If we can poke the cache and let the system reloads, it will much easier and quicker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619011#comment-13619011
]
rajeshbabu commented on HBASE-6469:
---
[~jxiang] We dont have watcher on these znodes. May be we can do this but it
will be little risky and difficult I feel.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8233) Poke the cache and let the system reload cached information
[ https://issues.apache.org/jira/browse/HBASE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619014#comment-13619014 ] Nick Dimiduk commented on HBASE-8233: - Isn't this resolved by the state consolidation proposed by [~sershe] in HBASE-7305? Poke the cache and let the system reload cached information --- Key: HBASE-8233 URL: https://issues.apache.org/jira/browse/HBASE-8233 Project: HBase Issue Type: New Feature Reporter: Jimmy Xiang Sometimes, cached information is stale and we don't have a way to get rid of it except restarting the server. If we can poke the cache and let the system reloads, it will much easier and quicker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7680) implement compaction policy for stripe compactions
[ https://issues.apache.org/jira/browse/HBASE-7680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7680: Attachment: HBASE-7680-v6-with-7679.patch HBASE-7680-v6.patch r feedback implement compaction policy for stripe compactions -- Key: HBASE-7680 URL: https://issues.apache.org/jira/browse/HBASE-7680 Project: HBase Issue Type: Sub-task Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Attachments: HBASE-7680-v0.patch, HBASE-7680-v0-with-7679-and-7935.patch, HBASE-7680-v1.patch, HBASE-7680-v1-with-7679.patch, HBASE-7680-v2.patch, HBASE-7680-v2-with-7679-and-8034.patch, HBASE-7680-v3.patch, HBASE-7680-v3-with-7679.patch, HBASE-7680-v4.patch, HBASE-7680-v4-with-7679.patch, HBASE-7680-v5.patch, HBASE-7680-v5-with-7679.patch, HBASE-7680-v6.patch, HBASE-7680-v6-with-7679.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[
https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619017#comment-13619017
]
stack commented on HBASE-8224:
--
The plugin examples are all about command-line. I tried to add a
'configuration' after adding plugin to the hadoop2 profile and got this:
{code}
[ERROR] The project org.apache.hbase:hbase:0.97.0-SNAPSHOT
(/Users/stack/checkouts/hbase/pom.xml) has 1 error
[ERROR] Malformed POM /Users/stack/checkouts/hbase/pom.xml: Unrecognised
tag: 'configuration' (position: START_TAG seen .../version\n
configuration... @1380:28) @ /Users/stack/checkouts/hbase/pom.xml, line
1380, column 28 - [Help 2]
{code}
... so it doesn't seem to like it.
Might be dodgy doing this automatically given it changes all poms and leaves
them changed... (this is all so dirty!) Thanks Enis.
Add '-hadoop1' or '-hadoop2' to our version string
--
Key: HBASE-8224
URL: https://issues.apache.org/jira/browse/HBASE-8224
Project: HBase
Issue Type: Task
Reporter: stack
Attachments: 8224-adding.classifiers.txt
So we can publish both the hadoop1 and the hadoop2 jars to a maven
repository, and so we can publish two packages, one for hadoop1 and one for
hadoop2, given how maven works, our only alternative (to the best of my
knowledge and after consulting others) is by amending the version string to
include hadoop1 or hadoop2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-7970:
Attachment: HBASE-7970-v1.patch
r feedback, fix javadoc
Improve file descriptor usage: currently, there are two file descriptors per
storefile
--
Key: HBASE-7970
URL: https://issues.apache.org/jira/browse/HBASE-7970
Project: HBase
Issue Type: Sub-task
Reporter: Himanshu Vashishtha
Assignee: Sergey Shelukhin
Attachments: HBASE-7970-v0.patch, HBASE-7970-v1.patch
This is because there are two open calls in the HFile: one with checksum and
another for without checksum support in v2:
see the method in HFile:createReaderWithEncoding()
{code}
FSDataInputStream fsdis = fs.open(path);
FSDataInputStream fsdisNoFsChecksum = fsdis;
// If the fs is not an instance of HFileSystem, then create an
// instance of HFileSystem that wraps over the specified fs.
// In this case, we will not be able to avoid checksumming inside
// the filesystem.
if (!(fs instanceof HFileSystem)) {
hfs = new HFileSystem(fs);
} else {
hfs = (HFileSystem)fs;
// open a stream to read data without checksum verification in
// the filesystem
fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8233) Poke the cache and let the system reload cached information
[ https://issues.apache.org/jira/browse/HBASE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619021#comment-13619021 ] Jimmy Xiang commented on HBASE-8233: Most likely not yet. Otherwise, HBASE-6469 should be closed. Poke the cache and let the system reload cached information --- Key: HBASE-8233 URL: https://issues.apache.org/jira/browse/HBASE-8233 Project: HBase Issue Type: New Feature Reporter: Jimmy Xiang Sometimes, cached information is stale and we don't have a way to get rid of it except restarting the server. If we can poke the cache and let the system reloads, it will much easier and quicker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619026#comment-13619026
]
Jeffrey Zhong commented on HBASE-8127:
--
[~rajesh23] I see. Sorry for the exercise. Let's go with 94_2.patch then. +1
from me.
Region of a disabling or disabled table could be stuck in transition state
when RS dies during Master initialization
Key: HBASE-8127
URL: https://issues.apache.org/jira/browse/HBASE-8127
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Jeffrey Zhong
Assignee: rajeshbabu
Fix For: 0.94.7
Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
The issue happens when a RS dies during a master starts up. After the RS
reports open to the new master instance and dies immediately thereafter, the
RITs of disabling tables(or disabled table) on the died RS will be in RIT
state forever.
I attached a patch to simulate the situation and you can run the following
command to reproduce the issue:
{code}mvn test -PlocalTests
-Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
Basically, we skip regions of a dead server inside
AM.processDeadServersAndRecoverLostRegions as the following code and relies
on SSH to process those skipped regions:
{code}
for (PairHRegionInfo, Result deadRegion : deadServer.getValue()) {
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
While in SSH, we skip regions of disabling(or disabled table) again by
function processDeadRegion. Finally comes to the issue that RITs of
disabling(or disabled table) stuck there forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619027#comment-13619027
]
Jimmy Xiang commented on HBASE-6469:
I agree. I think it is not needed. That's why I suggested HBASE-8233, which
may be useful in other scenarios too.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8225) [replication] minor code bug when registering ReplicationLogCleaner
[
https://issues.apache.org/jira/browse/HBASE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619030#comment-13619030
]
Jerry He commented on HBASE-8225:
-
Sergey,
Thanks! Are we ok with the patch? Since this is an internal code path fix, I
don't think we need a test or there is test that can test this?
[replication] minor code bug when registering ReplicationLogCleaner
---
Key: HBASE-8225
URL: https://issues.apache.org/jira/browse/HBASE-8225
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.94.6
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 0.95.0, 0.96.0, 0.94.7
Attachments: HBASE-8255-0.94.patch, HBASE-8255-trunk.patch
We try to register ReplicationLogCleaner by default. This is done by calling
Replication.decorateMasterConfiguration()in the master.
In the current Replication.decorateMasterConfiguration():
...
String plugins = conf.get(HBASE_MASTER_LOGCLEANER_PLUGINS);
if (!plugins.contains(ReplicationLogCleaner.class.toString())) {
conf.set(HBASE_MASTER_LOGCLEANER_PLUGINS,
plugins + , + ReplicationLogCleaner.class.getCanonicalName());
}
...
ReplicationLogCleaner.class.toString() will return prefix 'class' to the full
class name, which will make the if checking mostly useless.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8233) Poke the cache and let the system reload cached information
[ https://issues.apache.org/jira/browse/HBASE-8233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619032#comment-13619032 ] Nick Dimiduk commented on HBASE-8233: - No implemented, just proposed. Poke the cache and let the system reload cached information --- Key: HBASE-8233 URL: https://issues.apache.org/jira/browse/HBASE-8233 Project: HBase Issue Type: New Feature Reporter: Jimmy Xiang Sometimes, cached information is stale and we don't have a way to get rid of it except restarting the server. If we can poke the cache and let the system reloads, it will much easier and quicker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6469) Failure on enable/disable table will cause table state in zk to be left as enabling/disabling until master is restarted
[
https://issues.apache.org/jira/browse/HBASE-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619039#comment-13619039
]
rajeshbabu commented on HBASE-6469:
---
bq. That's why I suggested HBASE-8233, which may be useful in other scenarios
too.
But I didnt see any state mismatch with cache and znodes(in that case
HBASE-8233 may solve). May be I understood wrongly.
Failure on enable/disable table will cause table state in zk to be left as
enabling/disabling until master is restarted
---
Key: HBASE-6469
URL: https://issues.apache.org/jira/browse/HBASE-6469
Project: HBase
Issue Type: Bug
Affects Versions: 0.95.0, 0.94.6
Reporter: Enis Soztutar
Assignee: rajeshbabu
Fix For: 0.95.0, 0.94.7
Attachments: 6469-expose-force-r3.patch, HBASE-6469_2.patch,
HBASE-6469_3.patch, HBASE-6469_4.patch, HBASE-6469.patch,
HBASE-6469_retry_enable_or_disable.patch
In Enable/DisableTableHandler code, if something goes wrong in handling, the
table state in zk is left as ENABLING / DISABLING. After that we cannot force
any more action from the API or CLI, and the only recovery path is restarting
the master.
{code}
if (done) {
// Flip the table to enabled.
this.assignmentManager.getZKTable().setEnabledTable(
this.tableNameStr);
LOG.info(Table ' + this.tableNameStr
+ ' was successfully enabled. Status: done= + done);
} else {
LOG.warn(Table ' + this.tableNameStr
+ ' wasn't successfully enabled. Status: done= + done);
}
{code}
Here, if done is false, the table state is not changed. There is also no way
to set skipTableStateCheck from cli / api.
We have run into this issue a couple of times before.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8225) [replication] minor code bug when registering ReplicationLogCleaner
[
https://issues.apache.org/jira/browse/HBASE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619040#comment-13619040
]
stack commented on HBASE-8225:
--
+1 on commit
[replication] minor code bug when registering ReplicationLogCleaner
---
Key: HBASE-8225
URL: https://issues.apache.org/jira/browse/HBASE-8225
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.94.6
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 0.95.0, 0.96.0, 0.94.7
Attachments: HBASE-8255-0.94.patch, HBASE-8255-trunk.patch
We try to register ReplicationLogCleaner by default. This is done by calling
Replication.decorateMasterConfiguration()in the master.
In the current Replication.decorateMasterConfiguration():
...
String plugins = conf.get(HBASE_MASTER_LOGCLEANER_PLUGINS);
if (!plugins.contains(ReplicationLogCleaner.class.toString())) {
conf.set(HBASE_MASTER_LOGCLEANER_PLUGINS,
plugins + , + ReplicationLogCleaner.class.getCanonicalName());
}
...
ReplicationLogCleaner.class.toString() will return prefix 'class' to the full
class name, which will make the if checking mostly useless.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619061#comment-13619061
]
Ted Yu commented on HBASE-8127:
---
{code}
private ListHRegionInfo checkForDisablingOrDisabledTables(SetHRegionInfo
regionsFromRIT,
- ListHRegionInfo toAssign, RegionState rit, AssignmentManager
assignmentManager) {
-if (rit == null) {
- return toAssign;
+ ListHRegionInfo toAssign, RegionState rit, HRegionInfo hri,
{code}
Do we need to pass hri to the above method ? We can obtain this information
through rit, right ?
Region of a disabling or disabled table could be stuck in transition state
when RS dies during Master initialization
Key: HBASE-8127
URL: https://issues.apache.org/jira/browse/HBASE-8127
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: Jeffrey Zhong
Assignee: rajeshbabu
Fix For: 0.94.7
Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
reproduce-hang.patch
The issue happens when a RS dies during a master starts up. After the RS
reports open to the new master instance and dies immediately thereafter, the
RITs of disabling tables(or disabled table) on the died RS will be in RIT
state forever.
I attached a patch to simulate the situation and you can run the following
command to reproduce the issue:
{code}mvn test -PlocalTests
-Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
Basically, we skip regions of a dead server inside
AM.processDeadServersAndRecoverLostRegions as the following code and relies
on SSH to process those skipped regions:
{code}
for (PairHRegionInfo, Result deadRegion : deadServer.getValue()) {
nodes.remove(deadRegion.getFirst().getEncodedName());
}
{code}
While in SSH, we skip regions of disabling(or disabled table) again by
function processDeadRegion. Finally comes to the issue that RITs of
disabling(or disabled table) stuck there forever.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8236) Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase.
stack created HBASE-8236: Summary: Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase. Key: HBASE-8236 URL: https://issues.apache.org/jira/browse/HBASE-8236 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8236.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8236) Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase.
[ https://issues.apache.org/jira/browse/HBASE-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8236: - Attachment: 8236.txt Small patch Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase. --- Key: HBASE-8236 URL: https://issues.apache.org/jira/browse/HBASE-8236 Project: HBase Issue Type: Bug Reporter: stack Attachments: 8236.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-8236) Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase.
[ https://issues.apache.org/jira/browse/HBASE-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-8236. -- Resolution: Fixed Fix Version/s: 0.95.0 Assignee: stack Committed to 0.95 and trunk. Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase. --- Key: HBASE-8236 URL: https://issues.apache.org/jira/browse/HBASE-8236 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.95.0 Attachments: 8236.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-8237) Integrate HDFS request profiling with HBase request profiling
Alex Feinberg created HBASE-8237: Summary: Integrate HDFS request profiling with HBase request profiling Key: HBASE-8237 URL: https://issues.apache.org/jira/browse/HBASE-8237 Project: HBase Issue Type: New Feature Affects Versions: 0.89-fb Reporter: Alex Feinberg Assignee: Alex Feinberg Fix For: 0.89-fb Since the building blocks to retrieve the RegionServer/DataNode profiling data is done (in Facebook's HDFS branch -- the changes are/will be posted to Github soon), it would be great to integrate them together, so that the HBase client can not only get the RegionServer metrics but also the DataNode status. It will offer the client a much clear view from end to end perspective including the disk/network level detail information for each request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8219) Align Offline Merge with Online Merge
[ https://issues.apache.org/jira/browse/HBASE-8219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619075#comment-13619075 ] Matteo Bertozzi commented on HBASE-8219: A compaction is called on the merge region, this means that after that call the region will not have reference files, but the real data. right? so at this point the other two regions are removed by using the archiver, are they empty after the compaction? they should, right... maybe add an assert there. if what I have said above is true, I'm +1 on the patch (with the assert if you want) Align Offline Merge with Online Merge - Key: HBASE-8219 URL: https://issues.apache.org/jira/browse/HBASE-8219 Project: HBase Issue Type: Task Components: regionserver Affects Versions: 0.95.0 Reporter: Matteo Bertozzi Assignee: chunhui shen Attachments: hbase-8219v1.patch, hbase-8219v2.patch After HBASE-7403 we now have two different tools for online and offline merge, and the result produced by the two are different. (the online one works with snapshots, the offline not) We should remove the offline one, or align it to the online code. Most of the offline code in HRegion.merge() can be replaced with the one in RegionMergeTransaction, used by the online version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8133) avoid assign for disabling table regions(OPENING/PENDING_OPEN) in SSH
[
https://issues.apache.org/jira/browse/HBASE-8133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619085#comment-13619085
]
Hadoop QA commented on HBASE-8133:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576400/HBASE-8133_2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5081//console
This message is automatically generated.
avoid assign for disabling table regions(OPENING/PENDING_OPEN) in SSH
-
Key: HBASE-8133
URL: https://issues.apache.org/jira/browse/HBASE-8133
Project: HBase
Issue Type: Bug
Components: Region Assignment
Affects Versions: 0.95.0
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0
Attachments: HBASE-8133_2.patch, HBASE-8133.patch
Disabling table regions which are in PENDING_OPEN or OPENING on dead server
are getting assigned then unassiging.
We can avoid this by just mark OFFLINE for the regions,any way znodes of the
transitions got deleted as part of am.processServerShutdown(serverName).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8236) Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase.
[ https://issues.apache.org/jira/browse/HBASE-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-8236: - Attachment: version.txt Small addendum. Add the pom.version to the assembly tgz name. Also, exclude versionsBackup from rat check (these files are made by the versions plugin). Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase. --- Key: HBASE-8236 URL: https://issues.apache.org/jira/browse/HBASE-8236 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.95.0 Attachments: 8236.txt, version.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8236) Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase.
[ https://issues.apache.org/jira/browse/HBASE-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619090#comment-13619090 ] stack commented on HBASE-8236: -- Committed addendum to trunk and 0.95 branch. Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase. --- Key: HBASE-8236 URL: https://issues.apache.org/jira/browse/HBASE-8236 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.95.0 Attachments: 8236.txt, version.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619095#comment-13619095
]
Hadoop QA commented on HBASE-7970:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576415/HBASE-7970-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5082//console
This message is automatically generated.
Improve file descriptor usage: currently, there are two file descriptors per
storefile
--
Key: HBASE-7970
URL: https://issues.apache.org/jira/browse/HBASE-7970
Project: HBase
Issue Type: Sub-task
Reporter: Himanshu Vashishtha
Assignee: Sergey Shelukhin
Attachments: HBASE-7970-v0.patch, HBASE-7970-v1.patch
This is because there are two open calls in the HFile: one with checksum and
another for without checksum support in v2:
see the method in HFile:createReaderWithEncoding()
{code}
FSDataInputStream fsdis = fs.open(path);
FSDataInputStream fsdisNoFsChecksum = fsdis;
// If the fs is not an instance of HFileSystem, then create an
// instance of HFileSystem that wraps over the specified fs.
// In this case, we will not be able to avoid checksumming inside
// the filesystem.
if (!(fs instanceof HFileSystem)) {
hfs = new HFileSystem(fs);
} else {
hfs = (HFileSystem)fs;
// open a stream to read data without checksum verification in
// the filesystem
fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8028) Append, Increment: Adding rollback support
[ https://issues.apache.org/jira/browse/HBASE-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619104#comment-13619104 ] Jimmy Xiang commented on HBASE-8028: In MemStore#rollbackUpsert, internalAdd is used to add back the original value. Will this change the readpoint? Should we add it to snapshot as well? Append, Increment: Adding rollback support -- Key: HBASE-8028 URL: https://issues.apache.org/jira/browse/HBASE-8028 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.5 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.95.0 Attachments: HBase-8028-v1.patch, HBase-8028-v2.patch, HBase-8028-with-Increments-v1.patch, HBase-8028-with-Increments-v2.patch In case there is an exception while doing the log-sync, the memstore is not rollbacked, while the mvcc is _always_ forwarded to the writeentry created at the beginning of the operation. This may lead to scanners seeing results which are not synched to the fs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8213) global authorization may lose efficacy
[
https://issues.apache.org/jira/browse/HBASE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619121#comment-13619121
]
Hudson commented on HBASE-8213:
---
Integrated in hbase-0.95 #118 (See
[https://builds.apache.org/job/hbase-0.95/118/])
HBASE-8213. Global authorization may lose efficacy (Revision 1463196)
Result = SUCCESS
apurtell :
Files :
*
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java
*
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
global authorization may lose efficacy
---
Key: HBASE-8213
URL: https://issues.apache.org/jira/browse/HBASE-8213
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.95.0, 0.96.0, 0.94.6
Reporter: Jieshan Bean
Assignee: Jieshan Bean
Priority: Critical
Fix For: 0.95.0, 0.98.0, 0.94.7
Attachments: HBASE-8213-94.patch, HBASE-8213-trunk.patch
It depends on the order of which region be opened first.
Suppose we have one 1 regionserver and only 1 user region REGION-A on this
server, _acl_ region was on another regionserver. _acl_ was opened a few
seconds before REGION-A.
The global authorization data read from Zookeeper was overwritten by the data
read from configuration.
{code}
private TableAuthManager(ZooKeeperWatcher watcher, Configuration conf)
throws IOException {
this.conf = conf;
this.zkperms = new ZKPermissionWatcher(watcher, this, conf);
try {
// Read global authorization data from zookeeper.
this.zkperms.start();
} catch (KeeperException ke) {
LOG.error(ZooKeeper initialization failed, ke);
}
// It will overwrite globalCache.
// initialize global permissions based on configuration
globalCache = initGlobal(conf);
}
{code}
This issue can be easily reproduced by below steps:
1. Start a cluster with 3 regionservers.
2. Create a new table T1.
3. grant a new user USER-A with global authorization.
4. Kill 1 regionserver RS3 and switch balance off.
5. Start regionserver RS3.
6. Assign region T1 to RS3.
7. Put data with user USER-A.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8164) TestTableLockManager fails intermittently in trunk builds
[ https://issues.apache.org/jira/browse/HBASE-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619122#comment-13619122 ] Hudson commented on HBASE-8164: --- Integrated in hbase-0.95 #118 (See [https://builds.apache.org/job/hbase-0.95/118/]) HBASE-8164 TestTableLockManager fails intermittently in trunk builds (Revision 1463195) Result = SUCCESS stack : Files : * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestTableLockManager.java TestTableLockManager fails intermittently in trunk builds - Key: HBASE-8164 URL: https://issues.apache.org/jira/browse/HBASE-8164 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.95.0, 0.98.0 Attachments: 8164-addendum-2.txt, 8164-addendum.txt, 8164-v2.txt, 8164-v3.txt, 8164-v4.txt, HBASE-8164_addendum_3.patch, HBASE-8164_addendum_4.patch, HBASE-8164_addendum_5.patch In build #3979: testTableReadLock(org.apache.hadoop.hbase.master.TestTableLockManager): test timed out after 60 milliseconds -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8236) Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase.
[ https://issues.apache.org/jira/browse/HBASE-8236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619123#comment-13619123 ] Hudson commented on HBASE-8236: --- Integrated in hbase-0.95 #118 (See [https://builds.apache.org/job/hbase-0.95/118/]) HBASE-8236 Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase (Revision 1463259) Result = SUCCESS stack : Files : * /hbase/branches/0.95/hbase-assembly/pom.xml Set finalName property in hbase-assembly else basename is hbase-assembly rather than hbase. --- Key: HBASE-8236 URL: https://issues.apache.org/jira/browse/HBASE-8236 Project: HBase Issue Type: Bug Reporter: stack Assignee: stack Fix For: 0.95.0 Attachments: 8236.txt, version.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7680) implement compaction policy for stripe compactions
[
https://issues.apache.org/jira/browse/HBASE-7680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619140#comment-13619140
]
Hadoop QA commented on HBASE-7680:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576414/HBASE-7680-v6-with-7679.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 22 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5083//console
This message is automatically generated.
implement compaction policy for stripe compactions
--
Key: HBASE-7680
URL: https://issues.apache.org/jira/browse/HBASE-7680
Project: HBase
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HBASE-7680-v0.patch,
HBASE-7680-v0-with-7679-and-7935.patch, HBASE-7680-v1.patch,
HBASE-7680-v1-with-7679.patch, HBASE-7680-v2.patch,
HBASE-7680-v2-with-7679-and-8034.patch, HBASE-7680-v3.patch,
HBASE-7680-v3-with-7679.patch, HBASE-7680-v4.patch,
HBASE-7680-v4-with-7679.patch, HBASE-7680-v5.patch,
HBASE-7680-v5-with-7679.patch, HBASE-7680-v6.patch,
HBASE-7680-v6-with-7679.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8225) [replication] minor code bug when registering ReplicationLogCleaner
[
https://issues.apache.org/jira/browse/HBASE-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619147#comment-13619147
]
Sergey Shelukhin commented on HBASE-8225:
-
Committed to 95 and trunk. [~lhofhansl] any objections to 94?
[replication] minor code bug when registering ReplicationLogCleaner
---
Key: HBASE-8225
URL: https://issues.apache.org/jira/browse/HBASE-8225
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.94.6
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
Fix For: 0.95.0, 0.96.0, 0.94.7
Attachments: HBASE-8255-0.94.patch, HBASE-8255-trunk.patch
We try to register ReplicationLogCleaner by default. This is done by calling
Replication.decorateMasterConfiguration()in the master.
In the current Replication.decorateMasterConfiguration():
...
String plugins = conf.get(HBASE_MASTER_LOGCLEANER_PLUGINS);
if (!plugins.contains(ReplicationLogCleaner.class.toString())) {
conf.set(HBASE_MASTER_LOGCLEANER_PLUGINS,
plugins + , + ReplicationLogCleaner.class.getCanonicalName());
}
...
ReplicationLogCleaner.class.toString() will return prefix 'class' to the full
class name, which will make the if checking mostly useless.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7572) move metadata settings that duplicate xml config settings to CF/table config in a backward-compatible manner
[
https://issues.apache.org/jira/browse/HBASE-7572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619150#comment-13619150
]
Hadoop QA commented on HBASE-7572:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576410/HBASE-7572-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 18 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5084//console
This message is automatically generated.
move metadata settings that duplicate xml config settings to CF/table config
in a backward-compatible manner
Key: HBASE-7572
URL: https://issues.apache.org/jira/browse/HBASE-7572
Project: HBase
Issue Type: Sub-task
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HBASE-7572-v0.patch, HBASE-7572-v1.patch,
HBASE-7572-v2.patch, HBASE-7572-v3.patch
2nd part of splitting HBASE-7236
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619152#comment-13619152
]
Sergey Shelukhin commented on HBASE-7970:
-
[~mbertozzi] do you want to review? You have been involved with FS-abstraction
stuff. Thanks
Improve file descriptor usage: currently, there are two file descriptors per
storefile
--
Key: HBASE-7970
URL: https://issues.apache.org/jira/browse/HBASE-7970
Project: HBase
Issue Type: Sub-task
Reporter: Himanshu Vashishtha
Assignee: Sergey Shelukhin
Attachments: HBASE-7970-v0.patch, HBASE-7970-v1.patch
This is because there are two open calls in the HFile: one with checksum and
another for without checksum support in v2:
see the method in HFile:createReaderWithEncoding()
{code}
FSDataInputStream fsdis = fs.open(path);
FSDataInputStream fsdisNoFsChecksum = fsdis;
// If the fs is not an instance of HFileSystem, then create an
// instance of HFileSystem that wraps over the specified fs.
// In this case, we will not be able to avoid checksumming inside
// the filesystem.
if (!(fs instanceof HFileSystem)) {
hfs = new HFileSystem(fs);
} else {
hfs = (HFileSystem)fs;
// open a stream to read data without checksum verification in
// the filesystem
fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[
https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619155#comment-13619155
]
Andrew Purtell commented on HBASE-7904:
---
Does this change mean that coprocessors will have a different view of
configuration than the HRegion itself? I.e. not see updates to configuration
via CompoundConfiguration from the schema?
{noformat}
Index:
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
===
---
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(revision 1461235)
+++
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
(working copy)
@@ -484,7 +484,7 @@
this.rsAccounting = this.rsServices.getRegionServerAccounting();
// don't initialize coprocessors if not running within a regionserver
// TODO: revisit if coprocessors should load in other cases
- this.coprocessorHost = new RegionCoprocessorHost(this, rsServices, conf);
+ this.coprocessorHost = new RegionCoprocessorHost(this, rsServices,
baseConf);
this.metricsRegionWrapper = new MetricsRegionWrapperImpl(this);
this.metricsRegion = new MetricsRegion(this.metricsRegionWrapper);
} else {
{noformat}
Make mapreduce jobs pass based on 2.0.4-alpha
-
Key: HBASE-7904
URL: https://issues.apache.org/jira/browse/HBASE-7904
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt,
7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt,
7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt,
7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt,
7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt
2.0.3-alpha has been released.
We should upgrade the dependency.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-8224) Add '-hadoop1' or '-hadoop2' to our version string
[ https://issues.apache.org/jira/browse/HBASE-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619157#comment-13619157 ] Enis Soztutar commented on HBASE-8224: -- I've talked with one of our build guys, [~gkesavan] about this. He says that we should rather change the artifact name, instead of the version string. Add '-hadoop1' or '-hadoop2' to our version string -- Key: HBASE-8224 URL: https://issues.apache.org/jira/browse/HBASE-8224 Project: HBase Issue Type: Task Reporter: stack Attachments: 8224-adding.classifiers.txt So we can publish both the hadoop1 and the hadoop2 jars to a maven repository, and so we can publish two packages, one for hadoop1 and one for hadoop2, given how maven works, our only alternative (to the best of my knowledge and after consulting others) is by amending the version string to include hadoop1 or hadoop2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619158#comment-13619158 ] Andrew Purtell commented on HBASE-7904: --- Wait, I just saw the v9 patch has been committed to trunk and 0.95. Why is this JIRA still open? See my above comment. Should this be addressed as an addendum here or as a new JIRA? Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619159#comment-13619159 ] Siddharth Seth commented on HBASE-7904: --- Posted a summary at https://issues.apache.org/jira/browse/YARN-449?focusedCommentId=13619145page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13619145 bq. Did something change in the hadoop2 snapshot? A yarn patch got committed? Nothing that should've affected HBase (given the previous comment about clusters not running in parallel) bq. I suppose I could go look myself but I ask because supposedly there was an incompatibility issue. Do you happen to know what this incompatibility was (and against which version) ? bq. Regarding TestImportExport - I'm not sure why this test is using the HBaseTestingUtility differently than other tests. It was missing setting YARN conf in certain places. bq. We just want something that worked like minimr in hadoop1. We start the cluster do our tests, not in //, then shut down the cluster. I misunderstood you when you said // above. Yes we run our tests in //. We do not set up a minimr cluster once and run tests in // against it. I'm still not clear about this. (not so relevant now that the tests are passing, but would be nice to know). Yes we run our tests in // - this is a little confusing. As an example, can TestTableMapReduce run in parallel with TestHFileOutputFormat ?, or can multiple tests within TestTableMapReduce run in parallel, or something else. Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7415) [snapshots] Add task information to snapshot operation
[
https://issues.apache.org/jira/browse/HBASE-7415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619160#comment-13619160
]
Hadoop QA commented on HBASE-7415:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12576381/HBASE-7415-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.master.TestTableLockManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/5085//console
This message is automatically generated.
[snapshots] Add task information to snapshot operation
--
Key: HBASE-7415
URL: https://issues.apache.org/jira/browse/HBASE-7415
Project: HBase
Issue Type: New Feature
Components: Client, master, regionserver, snapshots, Zookeeper
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.95.0
Attachments: HBASE-7415-0.94.patch, HBASE-7415-0.94-v1.patch,
hbase-7415-v0.patch, hbase-7415-v1.patch, HBASE-7415-v1-rebase.patch,
HBASE-7415-v2.patch, HBASE-7415-v3.patch, HBASE-7415-v4.patch,
HBase_Snapshot_Task_UI.png
Snapshot operations should have some sort of progresss information available
via the WebUI so admins can track progress of operations. This should go a
long way to enable 'good' admins to not hose their clusters by running
concurrent snapshot operations (e.g. rename while a clone is in progress).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619162#comment-13619162 ] Andrew Purtell commented on HBASE-7904: --- Since I am the owner of coprocessors and this patch changes how RegionCoprocessorHost is instantiated, I would really have appreciated a ping. Thanks. Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619175#comment-13619175 ] Ted Yu commented on HBASE-7904: --- bq. Why is this JIRA still open? 0.95 should not ship with 2.0.4-SNAPSHOT build. That was why I left this open. If this JIRA is resolved, we can open another JIRA to update pom.xml when 2.0.4-ALPHA is released. bq. this patch changes how RegionCoprocessorHost is instantiated I exchanged a few emails with Nicolas who was the author of CompoundConfiguration. My initial plan was to allow CompoundConfiguration.setClass() to be called. He advised against that approach because CompoundConfiguration was designed to be immutable. I understand that the patch changed behavior for RegionCoprocessorHost instantiation. If there is a better approach to solving the issue of running mapreduce job on hadoop 2.0 *and* maintaining immutability of CompoundConfiguration, I would be more than happy to adopt. Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7970) Improve file descriptor usage: currently, there are two file descriptors per storefile
[
https://issues.apache.org/jira/browse/HBASE-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619177#comment-13619177
]
Matteo Bertozzi commented on HBASE-7970:
looks good to me, there's stuff like the use of volatile and a lock on the
stream but not on the other vars that I don't understand, maybe add a comment
on how many threads can access to the stream initialization calls.
Improve file descriptor usage: currently, there are two file descriptors per
storefile
--
Key: HBASE-7970
URL: https://issues.apache.org/jira/browse/HBASE-7970
Project: HBase
Issue Type: Sub-task
Reporter: Himanshu Vashishtha
Assignee: Sergey Shelukhin
Attachments: HBASE-7970-v0.patch, HBASE-7970-v1.patch
This is because there are two open calls in the HFile: one with checksum and
another for without checksum support in v2:
see the method in HFile:createReaderWithEncoding()
{code}
FSDataInputStream fsdis = fs.open(path);
FSDataInputStream fsdisNoFsChecksum = fsdis;
// If the fs is not an instance of HFileSystem, then create an
// instance of HFileSystem that wraps over the specified fs.
// In this case, we will not be able to avoid checksumming inside
// the filesystem.
if (!(fs instanceof HFileSystem)) {
hfs = new HFileSystem(fs);
} else {
hfs = (HFileSystem)fs;
// open a stream to read data without checksum verification in
// the filesystem
fsdisNoFsChecksum = hfs.getNoChecksumFs().open(path);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7904) Make mapreduce jobs pass based on 2.0.4-alpha
[ https://issues.apache.org/jira/browse/HBASE-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619180#comment-13619180 ] Andrew Purtell commented on HBASE-7904: --- bq. I understand that the patch changed behavior for RegionCoprocessorHost instantiation. So you are ok with coprocessors having a different configuration than HRegion? Don't you forsee issues with that? Make mapreduce jobs pass based on 2.0.4-alpha - Key: HBASE-7904 URL: https://issues.apache.org/jira/browse/HBASE-7904 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7904-addendum.txt, 7904.txt, 7904-v2-hadoop-2.0.txt, 7904-v2.txt, 7904-v4-hadoop-2.0.txt, 7904-v4.txt, 7904-v4.txt, 7904-v5-hadoop-2.0.txt, 7904-v5.txt, 7904-v6-hadoop-2.0.txt, 7904-v7-hadoop-2.0.txt, 7904-v8-hadoop-2.0.txt, 7904-v8.txt, 7904-v9-hadoop-2.0.txt, 7904-v9.txt, hbase-7904-v3.txt 2.0.3-alpha has been released. We should upgrade the dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
