[jira] [Updated] (HBASE-10999) Cross-row Transaction : Implement Percolator Algorithm on HBase
[ https://issues.apache.org/jira/browse/HBASE-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cuijianwei updated HBASE-10999: --- Description: Cross-row transaction is a desired function for database. It is not easy to keep ACID characteristics of cross-row transactions in distribute databases such as HBase, because data of cross-transaction might locate in different machines. In the paper http://research.google.com/pubs/pub36726.html, google presents an algorithm(named percolator) to implement cross-row transactions on BigTable. After analyzing the algorithm, we found percolator might also be a choice to provide cross-row transaction on HBase. The reasons includes: 1. Percolator could keep the ACID of cross-row transaction as described in google's paper. Percolator depends on a Global Incremental Timestamp Service to define the order of transactions, this is important to keep ACID of transaction. 2. Percolator algorithm could be totally implemented in client-side. This means we do not need to change the logic of server side. Users could easily include percolator in their client and adopt percolator APIs only when they want cross-row transaction. 3. Percolator is a general algorithm which could be implemented based on databases providing single-row transaction. Therefore, it is feasible to implement percolator on HBase. In last few months, we have implemented percolator on HBase, did correctness validation, performance test and finally successfully applied this algorithm in our production environment. Our works include: 1. percolator algorithm implementation on HBase. The current implementations includes: a). a Transaction module to provides put/delete/get/scan interfaces to do cross-row/cross-table transaction. b). a Global Incremental Timestamp Server to provide globally monotonically increasing timestamp for transaction. c). a LockCleaner module to resolve conflict when concurrent transactions mutate the same column. d). an internal module to implement prewrite/commit/get/scan logic of percolator. Although percolator logic could be totally implemented in client-side, we use coprocessor framework of HBase in our implementation. This is because coprocessor could provide percolator-specific Rpc interfaces such as prewrite/commit to reduce Rpc rounds and improve efficiency. Another reason to use coprocessor is that we want to decouple percolator's code from HBase so that users will get clean HBase code if they don't need cross-row transactions. In future, we will also explore the concurrent running characteristic of coprocessor to do cross-row mutations more efficiently. 2. an AccountTransfer simulation program to validate the correctness of implementation. This program will distribute initial values in different tables, rows and columns in HBase. Each column represents an account. Then, configured client threads will be concurrently started to read out a number of account values from different tables and rows by percolator's get; after this, clients will randomly transfer values among these accounts while keeping the sum unchanged, which simulates concurrent cross-table/cross-row transactions. To check the correctness of transactions, a checker thread will periodically scan account values from all columns, make sure the current total value is the same as the initial total value. We run this validation program while developing, this help us correct errors of implementation. 3. performance evaluation under various test situations. We compared percolator's APIs with HBase's with different data size and client thread count for single-column transaction which represents the worst performance case for percolator. We get the performance comparison result as (below): a) For read, the performance of percolator is 90% of HBase; b) For write, the performance of percolator is 23% of HBase. The drop derives from the overhead of percolator logic, the performance test result is similar as the result reported by google's paper. 4. Performance improvement. The write performance of percolator decreases more compared with HBase. This is because percolator's write needs to read data out to check write conflict and needs two Rpcs which do prewriting and commiting respectively. We are investigating ways to improve the write performance. We are glad to share current percolator implementation and hope this could provide a choice for users who want cross-row transactions because it does not need to change the code and logic of origin HBase. Comments and discussions are welcomed. was: Cross-row transaction is a desired function for database. It is not easy to keep ACID characteristics of cross-row transactions in distribute databases such as HBase, because data of cross-transaction might locate in different machines. In the paper http://research.google.com/pubs/pub36726.html, google
[jira] [Commented] (HBASE-10999) Cross-row Transaction : Implement Percolator Algorithm on HBase
[ https://issues.apache.org/jira/browse/HBASE-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204508#comment-14204508 ] cuijianwei commented on HBASE-10999: In last few months, we have updated Themis to achieve better performance and include more features: 1. Improve the single-row write performance from 23%(relative drop compared with HBase's put) to 60%(for most test cases). For single-row write transaction, we only write lock to MemStore in prewrite-phase, then, we erase correpsonding lock, write data and commit information to HLog in commit-phase. This won't break the correctness of percolator algorithm and will help improve the performance a lot for single-row write. 2. Support HBase 0.98. We create a branch: https://github.com/XiaoMi/themis/tree/for_hbase_0.98 to make themis support HBase 0.98(Currently, support HBase 0.98.5). All the functions of master branch will also be implemented in this branch. 3. Transaction TTL support and Old Data Clean. Users could set TTL for read/write transaction respectivley. Then, old data which could not be read will be cleaned periodly. 4. MapReduce Support. We implement ThemisTableInputFormat to scan data from themis-enable table in Map Job and ThemisTableOutputFormat to write data by themis transaction in Reducer Job. Mult-table scan and write are also supportted. 5. Implement Zookeeper based WorkerRegister. As mentioned in percolator paper, Running workers write a token into the Chubby lockservice, ZookeeperWorkerRegister implements this function and will help resolve conflict more efficiently. 6. Table Schema Support. Users could set THEMIS_ENABLE attribute to true to family which needs themis transaction, then, themis will automatically set corresponding attributes to the family and create lock family. For more details, please see: https://github.com/XiaoMi/themis (for HBase 0.94) and https://github.com/XiaoMi/themis/tree/for_hbase_0.98 (for HBase 0.98). Cross-row Transaction : Implement Percolator Algorithm on HBase --- Key: HBASE-10999 URL: https://issues.apache.org/jira/browse/HBASE-10999 Project: HBase Issue Type: New Feature Components: Transactions/MVCC Affects Versions: 0.99.0 Reporter: cuijianwei Assignee: cuijianwei Cross-row transaction is a desired function for database. It is not easy to keep ACID characteristics of cross-row transactions in distribute databases such as HBase, because data of cross-transaction might locate in different machines. In the paper http://research.google.com/pubs/pub36726.html, google presents an algorithm(named percolator) to implement cross-row transactions on BigTable. After analyzing the algorithm, we found percolator might also be a choice to provide cross-row transaction on HBase. The reasons includes: 1. Percolator could keep the ACID of cross-row transaction as described in google's paper. Percolator depends on a Global Incremental Timestamp Service to define the order of transactions, this is important to keep ACID of transaction. 2. Percolator algorithm could be totally implemented in client-side. This means we do not need to change the logic of server side. Users could easily include percolator in their client and adopt percolator APIs only when they want cross-row transaction. 3. Percolator is a general algorithm which could be implemented based on databases providing single-row transaction. Therefore, it is feasible to implement percolator on HBase. In last few months, we have implemented percolator on HBase, did correctness validation, performance test and finally successfully applied this algorithm in our production environment. Our works include: 1. percolator algorithm implementation on HBase. The current implementations includes: a). a Transaction module to provides put/delete/get/scan interfaces to do cross-row/cross-table transaction. b). a Global Incremental Timestamp Server to provide globally monotonically increasing timestamp for transaction. c). a LockCleaner module to resolve conflict when concurrent transactions mutate the same column. d). an internal module to implement prewrite/commit/get/scan logic of percolator. Although percolator logic could be totally implemented in client-side, we use coprocessor framework of HBase in our implementation. This is because coprocessor could provide percolator-specific Rpc interfaces such as prewrite/commit to reduce Rpc rounds and improve efficiency. Another reason to use coprocessor is that we want to decouple percolator's code from HBase so that users will get clean HBase code if they don't need cross-row transactions. In future, we will also explore the concurrent running characteristic of coprocessor
[jira] [Updated] (HBASE-10999) Cross-row Transaction : Implement Percolator Algorithm on HBase
[ https://issues.apache.org/jira/browse/HBASE-10999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cuijianwei updated HBASE-10999: --- Description: Cross-row transaction is a desired function for database. It is not easy to keep ACID characteristics of cross-row transactions in distribute databases such as HBase, because data of cross-transaction might locate in different machines. In the paper http://research.google.com/pubs/pub36726.html, google presents an algorithm(named percolator) to implement cross-row transactions on BigTable. After analyzing the algorithm, we found percolator might also be a choice to provide cross-row transaction on HBase. The reasons includes: 1. Percolator could keep the ACID of cross-row transaction as described in google's paper. Percolator depends on a Global Incremental Timestamp Service to define the order of transactions, this is important to keep ACID of transaction. 2. Percolator algorithm could be totally implemented in client-side. This means we do not need to change the logic of server side. Users could easily include percolator in their client and adopt percolator APIs only when they want cross-row transaction. 3. Percolator is a general algorithm which could be implemented based on databases providing single-row transaction. Therefore, it is feasible to implement percolator on HBase. In last few months, we have implemented percolator on HBase, did correctness validation, performance test and finally successfully applied this algorithm in our production environment. Our works include: 1. percolator algorithm implementation on HBase. The current implementations includes: a). a Transaction module to provides put/delete/get/scan interfaces to do cross-row/cross-table transaction. b). a Global Incremental Timestamp Server to provide globally monotonically increasing timestamp for transaction. c). a LockCleaner module to resolve conflict when concurrent transactions mutate the same column. d). an internal module to implement prewrite/commit/get/scan logic of percolator. Although percolator logic could be totally implemented in client-side, we use coprocessor framework of HBase in our implementation. This is because coprocessor could provide percolator-specific Rpc interfaces such as prewrite/commit to reduce Rpc rounds and improve efficiency. Another reason to use coprocessor is that we want to decouple percolator's code from HBase so that users will get clean HBase code if they don't need cross-row transactions. In future, we will also explore the concurrent running characteristic of coprocessor to do cross-row mutations more efficiently. 2. an AccountTransfer simulation program to validate the correctness of implementation. This program will distribute initial values in different tables, rows and columns in HBase. Each column represents an account. Then, configured client threads will be concurrently started to read out a number of account values from different tables and rows by percolator's get; after this, clients will randomly transfer values among these accounts while keeping the sum unchanged, which simulates concurrent cross-table/cross-row transactions. To check the correctness of transactions, a checker thread will periodically scan account values from all columns, make sure the current total value is the same as the initial total value. We run this validation program while developing, this help us correct errors of implementation. 3. performance evaluation under various test situations. We compared percolator's APIs with HBase's with different data size and client thread count for single-column transaction which represents the worst performance case for percolator. We get the performance comparison result as (below): a) For read, the performance of percolator is 85% of HBase; b) For write, the performance of percolator is 60% of HBase. The drop derives from the overhead of percolator logic, the read performance is about 10% lower compared to that reported in percolator paper(94% for percolator). The write performance is much better compared to that reported in percolator paper(23% for percolator). We improve the performance of single-column transaction(also for single-row transaction) by only writing MemStore in prewrite-phase which will reduce one time HLog's write. 4. MapReduce Support. We implement a group of classes to support read data by themis transaction in Mapper job and write data by themis transaction in Reduce job. 5. The master branch of themis(https://github.com/XiaoMi/themis) is based on HBase 0.94, we also create a branch(https://github.com/XiaoMi/themis/tree/for_hbase_0.98) to support hbase 0.98. We are glad to share current percolator implementation and hope this could provide a choice for users who want cross-row transactions because it does not need to change the code and logic of origin HBase.
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyunjiong updated HBASE-12444: - Attachment: hbase-12444.patch Use uint64 for total_number_of_requests. Total number of requests overflow because it's int -- Key: HBASE-12444 URL: https://issues.apache.org/jira/browse/HBASE-12444 Project: HBase Issue Type: Bug Components: hbck, master, regionserver Reporter: zhaoyunjiong Priority: Minor Attachments: hbase-12444.patch When running hbck, I noticed Number of requests was wrong: Average load: 466.41237113402065 Number of requests: -1835941345 Number of regions: 45242 Number of regions in transition: 0 The root cause is it use int, and clearly it overflowed. I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyunjiong updated HBASE-12444: - Hadoop Flags: Incompatible change Status: Patch Available (was: Open) Total number of requests overflow because it's int -- Key: HBASE-12444 URL: https://issues.apache.org/jira/browse/HBASE-12444 Project: HBase Issue Type: Bug Components: hbck, master, regionserver Reporter: zhaoyunjiong Priority: Minor Attachments: hbase-12444.patch When running hbck, I noticed Number of requests was wrong: Average load: 466.41237113402065 Number of requests: -1835941345 Number of regions: 45242 Number of regions in transition: 0 The root cause is it use int, and clearly it overflowed. I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[
https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204630#comment-14204630
]
Hadoop QA commented on HBASE-12444:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680559/hbase-12444.patch
against trunk revision .
ATTACHMENT ID: 12680559
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:[1094,15]
method setTotalNumberOfRequests in class
org.apache.hadoop.hbase.protobuf.generated.ClusterStatusProtos.ServerLoad.Builder
cannot be applied to given types;
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-server: Compilation failure
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java:[1094,15]
method setTotalNumberOfRequests in class
org.apache.hadoop.hbase.protobuf.generated.ClusterStatusProtos.ServerLoad.Builder
cannot be applied to given types;
[ERROR] required: int
[ERROR] found: long
[ERROR] reason: actual argument long cannot be converted to int by method
invocation conversion
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11625//console
This message is automatically generated.
Total number of requests overflow because it's int
--
Key: HBASE-12444
URL: https://issues.apache.org/jira/browse/HBASE-12444
Project: HBase
Issue Type: Bug
Components: hbck, master, regionserver
Reporter: zhaoyunjiong
Priority: Minor
Attachments: hbase-12444.patch
When running hbck, I noticed Number of requests was wrong:
Average load: 466.41237113402065
Number of requests: -1835941345
Number of regions: 45242
Number of regions in transition: 0
The root cause is it use int, and clearly it overflowed.
I'll update a patch later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-9817) Add throughput quota and enforce hard limits
[ https://issues.apache.org/jira/browse/HBASE-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Liangliang updated HBASE-9817: - Attachment: HBASE-9817.patch Add throughput quota and enforce hard limits Key: HBASE-9817 URL: https://issues.apache.org/jira/browse/HBASE-9817 Project: HBase Issue Type: New Feature Components: Coprocessors Reporter: He Liangliang Assignee: He Liangliang Attachments: HBASE-9817.patch There is planning to add region count and table count quota mentioned in HBASE-8410. However, it's also quite useful to control the requesting throughput inside the region server. For example, we don't want a data dumping task affecting the online services and it's better to enforce the throughput quota inside HBase. Another common scenario is multi-tenancy, i.e. a cluster is shared by multiple applications. The following rules will be supported: 1. per user quota limits the total read/write throughput initiated by a single user on any table. 2. per (user, table) quota limits the total read/write throughput initiated by a single user on a specified table. The implementation will use coprocessor to intercept and check each request on each region. And each region server allocate a portion of quota from the total specified quota (for that user or user + table) based on the portion of active regions (the whole cluster or the specified table) assigned on that region server. The request will be rejected or delayed if the limit is reached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[ https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12451: Summary: IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster (was: IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region split in rolling update of cluster) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster Key: HBASE-12451 URL: https://issues.apache.org/jira/browse/HBASE-12451 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minmum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[ https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12451: Description: Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minimum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ was: Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minmum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster Key: HBASE-12451 URL: https://issues.apache.org/jira/browse/HBASE-12451 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minimum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region split in rolling update of cluster
Liu Shaohui created HBASE-12451: --- Summary: IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region split in rolling update of cluster Key: HBASE-12451 URL: https://issues.apache.org/jira/browse/HBASE-12451 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minmum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12452) Add regular expression based split policy
He Liangliang created HBASE-12452: - Summary: Add regular expression based split policy Key: HBASE-12452 URL: https://issues.apache.org/jira/browse/HBASE-12452 Project: HBase Issue Type: Improvement Components: regionserver Reporter: He Liangliang Assignee: He Liangliang Priority: Minor The current DelimitedKeyPrefixRegionSplitPolicy split policy is not flexible enough to describe the split point prefix in some case. A regex based split policy is proposed, for example: ^[^\x00]+\x00[^\x00]+\x00 means the split point will always be truncated to a prefix at the second \0 char. We rely on this to implement local secondary index with data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12452) Add regular expression based split policy
[ https://issues.apache.org/jira/browse/HBASE-12452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Liangliang updated HBASE-12452: -- Attachment: HBASE-12452.patch Add regular expression based split policy - Key: HBASE-12452 URL: https://issues.apache.org/jira/browse/HBASE-12452 Project: HBase Issue Type: Improvement Components: regionserver Reporter: He Liangliang Assignee: He Liangliang Priority: Minor Attachments: HBASE-12452.patch The current DelimitedKeyPrefixRegionSplitPolicy split policy is not flexible enough to describe the split point prefix in some case. A regex based split policy is proposed, for example: ^[^\x00]+\x00[^\x00]+\x00 means the split point will always be truncated to a prefix at the second \0 char. We rely on this to implement local secondary index with data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12452) Add regular expression based split policy
[ https://issues.apache.org/jira/browse/HBASE-12452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Liangliang updated HBASE-12452: -- Status: Patch Available (was: Open) Add regular expression based split policy - Key: HBASE-12452 URL: https://issues.apache.org/jira/browse/HBASE-12452 Project: HBase Issue Type: Improvement Components: regionserver Reporter: He Liangliang Assignee: He Liangliang Priority: Minor Attachments: HBASE-12452.patch The current DelimitedKeyPrefixRegionSplitPolicy split policy is not flexible enough to describe the split point prefix in some case. A regex based split policy is proposed, for example: ^[^\x00]+\x00[^\x00]+\x00 means the split point will always be truncated to a prefix at the second \0 char. We rely on this to implement local secondary index with data types. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[ https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204741#comment-14204741 ] zhangduo commented on HBASE-12451: -- Setting minimum split size will delay the first split operation which is good for load balancing(a new table will have only one region if not pre-split) What about reduce the boost factor(current is region_count^3) and use the total region count of this table(I do not know where is it stored but I think we can get it somewhere such as master or zookeeper) instead of the region count of this table on this server? IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster Key: HBASE-12451 URL: https://issues.apache.org/jira/browse/HBASE-12451 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minimum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204806#comment-14204806 ] zhaoyunjiong commented on HBASE-12444: -- Should I need include below two generated files into patch? hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterStatusProtos.java hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/protobuf/generated/StorageClusterStatusMessage.java Total number of requests overflow because it's int -- Key: HBASE-12444 URL: https://issues.apache.org/jira/browse/HBASE-12444 Project: HBase Issue Type: Bug Components: hbck, master, regionserver Reporter: zhaoyunjiong Priority: Minor Attachments: hbase-12444.patch When running hbck, I noticed Number of requests was wrong: Average load: 466.41237113402065 Number of requests: -1835941345 Number of regions: 45242 Number of regions in transition: 0 The root cause is it use int, and clearly it overflowed. I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12452) Add regular expression based split policy
[
https://issues.apache.org/jira/browse/HBASE-12452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204825#comment-14204825
]
Hadoop QA commented on HBASE-12452:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680573/HBASE-12452.patch
against trunk revision .
ATTACHMENT ID: 12680573
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3785 checkstyle errors (more than the trunk's current 3783 errors).
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1 release
audit warnings (more than the trunk's current 0 warnings).
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/patchReleaseAuditWarnings.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11626//console
This message is automatically generated.
Add regular expression based split policy
-
Key: HBASE-12452
URL: https://issues.apache.org/jira/browse/HBASE-12452
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: He Liangliang
Assignee: He Liangliang
Priority: Minor
Attachments: HBASE-12452.patch
The current DelimitedKeyPrefixRegionSplitPolicy split policy is not flexible
enough to describe the split point prefix in some case. A regex based split
policy is proposed, for example:
^[^\x00]+\x00[^\x00]+\x00
means the split point will always be truncated to a prefix at the second \0
char. We rely on this to implement local secondary index with data types.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14204958#comment-14204958
]
sri commented on HBASE-12445:
-
In the ProtoBufUtil.toMutation(final MutationType type, final Mutation
mutation, MutationProto.Builder builder, long nonce)
QualifierValue.Builder valueBuilder = QualifierValue.newBuilder(); at line
number 1110 should be inside the inner for loop at 1115 or needs to clear the
ValueBuilder. Otherwise valueBuilder is carrying over the previous values that
where set. In this case the DeleteType is getting carried over to the next
columns and deleting all the columns.
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Attachments: TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12453) Make region available once it's open
Jimmy Xiang created HBASE-12453: --- Summary: Make region available once it's open Key: HBASE-12453 URL: https://issues.apache.org/jira/browse/HBASE-12453 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Currently (in trunk, with zk-less assignment), a region is available to serving requests only after RS notifies the master the region is open, and the meta is updated with the new location. We may be able to do better than this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hani Nadra updated HBASE-12445:
---
Attachment: 0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch
Here the patch with the updated test and the fix suggest by sri in 1178:
QualifierValue.Builder valueBuilder = QualifierValue.newBuilder(); at line
number 1110 should be inside the inner for loop at 1115. Otherwise valueBuilder
is carrying over the previous values that where set. In this case the
DeleteType is getting carried over to the next columns and deleting all the
columns. Attached the testcase to the defect.Could you please reopen the defect.
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12445:
---
Status: Patch Available (was: Open)
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12445:
---
Attachment: 12445-v2.patch
Slightly modified patch where valueBuilder is cleared inside the loop.
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205035#comment-14205035 ] Ted Yu commented on HBASE-12444: Please include generated files for QA to run tests. Total number of requests overflow because it's int -- Key: HBASE-12444 URL: https://issues.apache.org/jira/browse/HBASE-12444 Project: HBase Issue Type: Bug Components: hbck, master, regionserver Reporter: zhaoyunjiong Priority: Minor Attachments: hbase-12444.patch When running hbck, I noticed Number of requests was wrong: Average load: 466.41237113402065 Number of requests: -1835941345 Number of regions: 45242 Number of regions in transition: 0 The root cause is it use int, and clearly it overflowed. I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205158#comment-14205158
]
Hadoop QA commented on HBASE-12445:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680604/12445-v2.patch
against trunk revision .
ATTACHMENT ID: 12680604
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11627//console
This message is automatically generated.
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205198#comment-14205198
]
Ted Yu commented on HBASE-12445:
[~hnadra]:
Should this issue be assigned to sri ?
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12445:
---
Fix Version/s: 0.99.2
0.98.9
2.0.0
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205269#comment-14205269 ] Elliott Clark commented on HBASE-12444: --- This will break compatibility of rolling restarts. Total number of requests overflow because it's int -- Key: HBASE-12444 URL: https://issues.apache.org/jira/browse/HBASE-12444 Project: HBase Issue Type: Bug Components: hbck, master, regionserver Reporter: zhaoyunjiong Priority: Minor Attachments: hbase-12444.patch When running hbck, I noticed Number of requests was wrong: Average load: 466.41237113402065 Number of requests: -1835941345 Number of regions: 45242 Number of regions in transition: 0 The root cause is it use int, and clearly it overflowed. I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205281#comment-14205281
]
sri commented on HBASE-12445:
-
Hani and I are collegues
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
Andrew Purtell created HBASE-12454:
--
Summary: Setting didPerformCompaction early in HRegion#compact
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last finally
block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1565,8 +1565,8 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver { //
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205291#comment-14205291
]
Ted Yu commented on HBASE-12445:
Understood.
You two can decide who the assignee is.
I can put both your names when committing
Thanks
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205296#comment-14205296
]
Andrew Purtell commented on HBASE-12454:
Is this right or am I missing something?
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1565,8 +1565,8 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver { //
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Attachment: HBASE-12454-0.98.patch
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1565,8 +1565,8 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver { //
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Attachment: HBASE-12454.patch
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1565,8 +1565,8 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver { //
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Status: Patch Available (was: Open)
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1565,8 +1565,8 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver { //
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12343) Document recommended configuration for 0.98 from HBASE-11964
[ https://issues.apache.org/jira/browse/HBASE-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12343: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master Document recommended configuration for 0.98 from HBASE-11964 Key: HBASE-12343 URL: https://issues.apache.org/jira/browse/HBASE-12343 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 2.0.0 Attachments: HBASE-12343.patch We're not committing the configuration changes from HBASE-11964 to 0.98 but they should be the recommend configuration for replication. Add a paragraph to the replication section of the manual on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Description:
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last finally
block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in +
this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
was:
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last finally
block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1565,8 +1565,8 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver { //
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
{code}
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205339#comment-14205339
]
Ted Yu commented on HBASE-12445:
[~apurtell][~enis]:
Want to take a look ?
Thanks
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205375#comment-14205375
]
Hani Nadra commented on HBASE-12445:
Hi Ted,
If it just a matter of adding the name to the code, sri can be the assignee.
Please let us know if anything further is needed to get the patch applied.
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205389#comment-14205389
]
Lars Hofhansl commented on HBASE-12454:
---
That looks right to me. Too bad the RC is out already.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12319) Inconsistencies during region recovery due to close/open of a region during recovery
[
https://issues.apache.org/jira/browse/HBASE-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205404#comment-14205404
]
Andrew Purtell commented on HBASE-12319:
+1, let's commit this
Inconsistencies during region recovery due to close/open of a region during
recovery
Key: HBASE-12319
URL: https://issues.apache.org/jira/browse/HBASE-12319
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.7, 0.99.1
Reporter: Devaraj Das
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments: HBASE-12319-v2.patch, HBASE-12319.patch
In one of my test runs, I saw the following:
{noformat}
2014-10-14 13:45:30,782 DEBUG
[StoreOpener-51af4bd23dc32a940ad2dd5435f00e1d-1] regionserver.HStore: loaded
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/test_cf/d6df5cfe15ca41d68c619489fbde4d04,
isReference=false, isBulkLoadResult=false, seqid=141197, majorCompaction=true
2014-10-14 13:45:30,788 DEBUG [RS_OPEN_REGION-hor9n01:60020-1]
regionserver.HRegion: Found 3 recovered edits file(s) under
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d
.
.
2014-10-14 13:45:31,916 WARN [RS_OPEN_REGION-hor9n01:60020-1]
regionserver.HRegion: Null or non-existent edits file:
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/recovered.edits/0198080
{noformat}
The above logs is from a regionserver, say RS2. From the initial analysis it
seemed like the master asked a certain regionserver to open the region (let's
say RS1) and for some reason asked it to close soon after. The open was still
proceeding on RS1 but the master reassigned the region to RS2. This also
started the recovery but it ended up seeing an inconsistent view of the
recovered-edits files (it reports missing files as per the logs above) since
the first regionserver (RS1) deleted some files after it completed the
recovery. When RS2 really opens the region, it might not see the recent data
that was written by flushes on hor9n10 during the recovery process. Reads of
that data would have inconsistencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205407#comment-14205407
]
Andrew Purtell commented on HBASE-12454:
I'm going to veto the RC for this and since HBASE-12319 didn't quite make it
but was marked critical
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205408#comment-14205408
]
Andrew Purtell commented on HBASE-12454:
Will commit this shortly to 0.98+ unless there is an objection
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12319) Inconsistencies during region recovery due to close/open of a region during recovery
[
https://issues.apache.org/jira/browse/HBASE-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205411#comment-14205411
]
Andrew Purtell commented on HBASE-12319:
Will commit this shortly unless there is an objection
Inconsistencies during region recovery due to close/open of a region during
recovery
Key: HBASE-12319
URL: https://issues.apache.org/jira/browse/HBASE-12319
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.7, 0.99.1
Reporter: Devaraj Das
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments: HBASE-12319-v2.patch, HBASE-12319.patch
In one of my test runs, I saw the following:
{noformat}
2014-10-14 13:45:30,782 DEBUG
[StoreOpener-51af4bd23dc32a940ad2dd5435f00e1d-1] regionserver.HStore: loaded
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/test_cf/d6df5cfe15ca41d68c619489fbde4d04,
isReference=false, isBulkLoadResult=false, seqid=141197, majorCompaction=true
2014-10-14 13:45:30,788 DEBUG [RS_OPEN_REGION-hor9n01:60020-1]
regionserver.HRegion: Found 3 recovered edits file(s) under
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d
.
.
2014-10-14 13:45:31,916 WARN [RS_OPEN_REGION-hor9n01:60020-1]
regionserver.HRegion: Null or non-existent edits file:
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/recovered.edits/0198080
{noformat}
The above logs is from a regionserver, say RS2. From the initial analysis it
seemed like the master asked a certain regionserver to open the region (let's
say RS1) and for some reason asked it to close soon after. The open was still
proceeding on RS1 but the master reassigned the region to RS2. This also
started the recovery but it ended up seeing an inconsistent view of the
recovered-edits files (it reports missing files as per the logs above) since
the first regionserver (RS1) deleted some files after it completed the
recovery. When RS2 really opens the region, it might not see the recent data
that was written by flushes on hor9n10 during the recovery process. Reads of
that data would have inconsistencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12445:
---
Fix Version/s: (was: 0.98.9)
0.98.8
Hadoop Flags: Reviewed
Do you want this in the next 0.98.8 RC [~apurtell] ?
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11964) Improve spreading replication load from failed regionservers
[ https://issues.apache.org/jira/browse/HBASE-11964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205428#comment-14205428 ] Hudson commented on HBASE-11964: SUCCESS: Integrated in HBase-TRUNK #5757 (See [https://builds.apache.org/job/HBase-TRUNK/5757/]) HBASE-12343 Document recommended configuration for 0.98 from HBASE-11964 (apurtell: rev 724b4a4693138e518e420bee1dfef3b69b0d0642) * src/main/docbkx/ops_mgt.xml Improve spreading replication load from failed regionservers Key: HBASE-11964 URL: https://issues.apache.org/jira/browse/HBASE-11964 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 2.0.0, 0.99.2 Attachments: HBASE-11964.patch, HBASE-11964.patch, HBASE-11964.patch Improve replication source thread handling. Improve fanout when transferring queues. Ensure replication sources terminate properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12343) Document recommended configuration for 0.98 from HBASE-11964
[ https://issues.apache.org/jira/browse/HBASE-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205429#comment-14205429 ] Hudson commented on HBASE-12343: SUCCESS: Integrated in HBase-TRUNK #5757 (See [https://builds.apache.org/job/HBase-TRUNK/5757/]) HBASE-12343 Document recommended configuration for 0.98 from HBASE-11964 (apurtell: rev 724b4a4693138e518e420bee1dfef3b69b0d0642) * src/main/docbkx/ops_mgt.xml Document recommended configuration for 0.98 from HBASE-11964 Key: HBASE-12343 URL: https://issues.apache.org/jira/browse/HBASE-12343 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 2.0.0 Attachments: HBASE-12343.patch We're not committing the configuration changes from HBASE-11964 to 0.98 but they should be the recommend configuration for replication. Add a paragraph to the replication section of the manual on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9817) Add throughput quota and enforce hard limits
[ https://issues.apache.org/jira/browse/HBASE-9817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205430#comment-14205430 ] Jonathan Hsieh commented on HBASE-9817: --- I think HBASE-11598 covers the same functionality and has been committed by [~mbertozzi]. Do you guys agree? Add throughput quota and enforce hard limits Key: HBASE-9817 URL: https://issues.apache.org/jira/browse/HBASE-9817 Project: HBase Issue Type: New Feature Components: Coprocessors Reporter: He Liangliang Assignee: He Liangliang Attachments: HBASE-9817.patch There is planning to add region count and table count quota mentioned in HBASE-8410. However, it's also quite useful to control the requesting throughput inside the region server. For example, we don't want a data dumping task affecting the online services and it's better to enforce the throughput quota inside HBase. Another common scenario is multi-tenancy, i.e. a cluster is shared by multiple applications. The following rules will be supported: 1. per user quota limits the total read/write throughput initiated by a single user on any table. 2. per (user, table) quota limits the total read/write throughput initiated by a single user on a specified table. The implementation will use coprocessor to intercept and check each request on each region. And each region server allocate a portion of quota from the total specified quota (for that user or user + table) based on the portion of active regions (the whole cluster or the specified table) assigned on that region server. The request will be rejected or delayed if the limit is reached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205433#comment-14205433
]
Hadoop QA commented on HBASE-12454:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680650/HBASE-12454.patch
against trunk revision .
ATTACHMENT ID: 12680650
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestCompaction
org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11628//console
This message is automatically generated.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205483#comment-14205483
]
Andrew Purtell commented on HBASE-12454:
Those are interesting test failures. They didn't fail locally but seem highly
relevant. Looking into it.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Assignee: Andrew Purtell
Status: Open (was: Patch Available)
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205509#comment-14205509
]
Lars Hofhansl commented on HBASE-12454:
---
Hmm... Yes. Maybe put it in the next RC, then.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205537#comment-14205537
]
Andrew Purtell commented on HBASE-12454:
Maybe.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-10483) Provide API for retrieving info port when hbase.master.info.port is set to 0
[ https://issues.apache.org/jira/browse/HBASE-10483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205633#comment-14205633 ] Ted Yu commented on HBASE-10483: Put some minor comments on review board. Please submit to Hadoop QA for test run. Provide API for retrieving info port when hbase.master.info.port is set to 0 Key: HBASE-10483 URL: https://issues.apache.org/jira/browse/HBASE-10483 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Liu Shaohui Attachments: HBASE-10483-trunk-v1.diff, HBASE-10483-trunk-v2.diff, HBASE-10483-v3.diff When hbase.master.info.port is set to 0, info port is dynamically determined. An API should be provided so that client can retrieve the actual info port. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Attachment: HBASE-12454.patch
It's a test only accounting problem.
The failure is an assert in HRegion#reportCompactionRequestEnd now that we are
actually calling HStore#cancelRequestedCompaction when interrupted. However the
test goes to HRegion#compactStores directly so the compaction request start is
not first reported with HRegion#reportCompactionRequestStart. I wasn't running
with assertions enabled before locally but am now.
Updated patch.
Patch also includes a fix for a NPE from TestCompaction that the debugger
caught that is normally silent.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch,
HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Fix Version/s: 0.98.8
0.99.2
2.0.0
Status: Patch Available (was: Open)
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.99.2, 0.98.8
Attachments: HBASE-12454-0.98.patch, HBASE-12454.patch,
HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12445:
---
Fix Version/s: (was: 0.98.8)
0.98.9
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12445) hbase is removing all remaining cells immediately after the cell marked with marker = KeyValue.Type.DeleteColumn via PUT
[
https://issues.apache.org/jira/browse/HBASE-12445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205663#comment-14205663
]
Andrew Purtell commented on HBASE-12445:
No, let's put it in for 0.98.9-SNAPSHOT
hbase is removing all remaining cells immediately after the cell marked with
marker = KeyValue.Type.DeleteColumn via PUT
Key: HBASE-12445
URL: https://issues.apache.org/jira/browse/HBASE-12445
Project: HBase
Issue Type: Bug
Reporter: sri
Fix For: 2.0.0, 0.98.9, 0.99.2
Attachments:
0001-HBASE-11788-hbase-is-not-deleting-the-cell-when-a-Pu.patch,
12445-v2.patch, TestPutAfterDeleteColumn.java
Code executed:
{code}
@Test
public void testHbasePutDeleteCell() throws Exception {
TableName tableName = TableName.valueOf(my_test);
Configuration configuration = HBaseConfiguration.create();
HTableInterface table = new HTable(configuration, tableName);
final String rowKey = 12345;
final byte[] familly = Bytes.toBytes(default);
// put one row
Put put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b));
put.add(family, Bytes.toBytes(C), Bytes.toBytes(c));
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d));
table.put(put);
// get row back and assert the values
Get get = new Get(rowKey);
Result result = table.get(get);
assertTrue(Column A value should be a,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a));
assertTrue(Column B value should be b,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b));
assertTrue(Column C value should be c,
Bytes.toString(result.getValue(family,
Bytes.toBytes(C))).equals(c));
assertTrue(Column D value should be d,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d));
// put the same row again with C column deleted
put = new Put(rowKey);
put.add(family, Bytes.toBytes(A), Bytes.toBytes(a1));
put.add(family, Bytes.toBytes(B), Bytes.toBytes(b1));
KeyValue marker = new KeyValue(rowKey, family, Bytes.toBytes(C),
HConstants.LATEST_TIMESTAMP, KeyValue.Type.DeleteColumn);
put.add(marker);
put.add(family, Bytes.toBytes(D), Bytes.toBytes(d1));
table.put(put);
// get row back and assert the values
get = new Get(rowKey);
result = table.get(get);
assertTrue(Column A value should be a1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(A))).equals(a1));
assertTrue(Column B value should be b1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(B))).equals(b1));
assertTrue(Column C should not exist,
result.getValue(family, Bytes.toBytes(C)) == null);
assertTrue(Column D value should be d1,
Bytes.toString(result.getValue(family,
Bytes.toBytes(D))).equals(d1));
}
{code}
This assertion fails, the cell D is also deleted
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12319) Inconsistencies during region recovery due to close/open of a region during recovery
[
https://issues.apache.org/jira/browse/HBASE-12319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12319:
---
Fix Version/s: (was: 0.98.9)
0.98.8
Inconsistencies during region recovery due to close/open of a region during
recovery
Key: HBASE-12319
URL: https://issues.apache.org/jira/browse/HBASE-12319
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.7, 0.99.1
Reporter: Devaraj Das
Assignee: Jeffrey Zhong
Priority: Critical
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12319-v2.patch, HBASE-12319.patch
In one of my test runs, I saw the following:
{noformat}
2014-10-14 13:45:30,782 DEBUG
[StoreOpener-51af4bd23dc32a940ad2dd5435f00e1d-1] regionserver.HStore: loaded
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/test_cf/d6df5cfe15ca41d68c619489fbde4d04,
isReference=false, isBulkLoadResult=false, seqid=141197, majorCompaction=true
2014-10-14 13:45:30,788 DEBUG [RS_OPEN_REGION-hor9n01:60020-1]
regionserver.HRegion: Found 3 recovered edits file(s) under
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d
.
.
2014-10-14 13:45:31,916 WARN [RS_OPEN_REGION-hor9n01:60020-1]
regionserver.HRegion: Null or non-existent edits file:
hdfs://hor9n01.gq1.ygridcore.net:8020/apps/hbase/data/data/default/IntegrationTestIngest/51af4bd23dc32a940ad2dd5435f00e1d/recovered.edits/0198080
{noformat}
The above logs is from a regionserver, say RS2. From the initial analysis it
seemed like the master asked a certain regionserver to open the region (let's
say RS1) and for some reason asked it to close soon after. The open was still
proceeding on RS1 but the master reassigned the region to RS2. This also
started the recovery but it ended up seeing an inconsistent view of the
recovered-edits files (it reports missing files as per the logs above) since
the first regionserver (RS1) deleted some files after it completed the
recovery. When RS2 really opens the region, it might not see the recent data
that was written by flushes on hor9n10 during the recovery process. Reads of
that data would have inconsistencies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-12454:
---
Attachment: (was: HBASE-12454-0.98.patch)
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12454.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12279) Generated thrift files were generated with the wrong parameters
[
https://issues.apache.org/jira/browse/HBASE-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205668#comment-14205668
]
Andrew Purtell commented on HBASE-12279:
Do you want this in 0.94 [~lhofhansl]?
Generated thrift files were generated with the wrong parameters
---
Key: HBASE-12279
URL: https://issues.apache.org/jira/browse/HBASE-12279
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0, 0.98.0, 0.99.0
Reporter: Niels Basjes
Assignee: Niels Basjes
Fix For: 2.0.0, 0.98.8, 0.94.26, 0.99.2
Attachments: HBASE-12279-2014-10-16-v1.patch,
HBASE-12279-2014-11-07-v2.patch
It turns out that the java code generated from the thrift files have been
generated with the wrong settings.
Instead of the documented
([thrift|http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/thrift/package-summary.html],
[thrift2|http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/thrift2/package-summary.html])
{code}
thrift -strict --gen java:hashcode
{code}
the current files seem to be generated instead with
{code}
thrift -strict --gen java
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205679#comment-14205679
]
Lars Hofhansl commented on HBASE-12454:
---
Looks good. +1
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12454.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205706#comment-14205706
]
Andrew Purtell commented on HBASE-12454:
Waiting for Jenkins to come around again and give this another shot
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12454.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Yuan Jiang updated HBASE-11099:
---
Assignee: Stephen Yuan Jiang
Two situations where we could open a region with smaller sequence number
Key: HBASE-11099
URL: https://issues.apache.org/jira/browse/HBASE-11099
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.99.0
Reporter: Jeffrey Zhong
Assignee: Stephen Yuan Jiang
Fix For: 0.99.2
Recently I happened to run into code where we potentially could open region
with smaller sequence number:
1) Inside function: HRegion#internalFlushcache. This is due to we change the
way WAL Sync where we use late binding(assign sequence number right before
wal sync).
The flushSeqId may less than the change sequence number included in the flush
which may cause later region opening code to use a smaller than expected
sequence number when we reopen the region.
{code}
flushSeqId = this.sequenceId.incrementAndGet();
...
mvcc.waitForRead(w);
{code}
2) HRegion#replayRecoveredEdits where we have following code:
{code}
...
if (coprocessorHost != null) {
status.setStatus(Running pre-WAL-restore hook in coprocessors);
if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
val)) {
// if bypass this log entry, ignore it ...
continue;
}
}
...
currentEditSeqId = key.getLogSeqNum();
{code}
If coprocessor skip some tail WALEdits, then the function will return smaller
currentEditSeqId. In the end, a region may also open with a smaller sequence
number. This may cause data loss because Master may record a larger flushed
sequence Id and some WALEdits maybe skipped during recovery if the region
fail again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12444) Total number of requests overflow because it's int
[ https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyunjiong updated HBASE-12444: - Attachment: hbase-12444-v1.patch Update patch, include two generated files. Total number of requests overflow because it's int -- Key: HBASE-12444 URL: https://issues.apache.org/jira/browse/HBASE-12444 Project: HBase Issue Type: Bug Components: hbck, master, regionserver Reporter: zhaoyunjiong Priority: Minor Attachments: hbase-12444-v1.patch, hbase-12444.patch When running hbck, I noticed Number of requests was wrong: Average load: 466.41237113402065 Number of requests: -1835941345 Number of regions: 45242 Number of regions in transition: 0 The root cause is it use int, and clearly it overflowed. I'll update a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11979) Compaction progress reporting is wrong
[
https://issues.apache.org/jira/browse/HBASE-11979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205792#comment-14205792
]
Stephen Yuan Jiang commented on HBASE-11979:
[~esteban] are you actively working on it? One customer recently hit this
issue and it would be nice to get this fixed in 1.0.0.
Compaction progress reporting is wrong
--
Key: HBASE-11979
URL: https://issues.apache.org/jira/browse/HBASE-11979
Project: HBase
Issue Type: Bug
Reporter: Andrew Purtell
Assignee: Esteban Gutierrez
Priority: Minor
Fix For: 2.0.0, 0.98.9, 0.99.2
This is a long standing problem and previously could be observed in
regionserver metrics, but, we recently added logging for long running
compactions, and this has exposed the issue in a new way, e.g.
{noformat}
2014-09-15 14:20:59,450 DEBUG
[regionserver8120-largeCompactions-1410813534627]
compactions.Compactor: Compaction progress: 22683625/6808179 (333.18%),
rate=162.08 kB/sec
{noformat}
The 'rate' reported in such logging is consistent and what we were really
after, but the progress indication is clearly broken and should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12332) [mob] use filelink instad of retry when resolving an hfilelink.
[ https://issues.apache.org/jira/browse/HBASE-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205814#comment-14205814 ] Jingcheng Du commented on HBASE-12332: -- Hi Jon, [~jmhsieh], do you have comments on this patch? Thanks. [mob] use filelink instad of retry when resolving an hfilelink. --- Key: HBASE-12332 URL: https://issues.apache.org/jira/browse/HBASE-12332 Project: HBase Issue Type: Sub-task Components: mob Affects Versions: hbase-11339 Reporter: Jonathan Hsieh Fix For: hbase-11339 Attachments: HBASE-12332-V1.diff in the snapshot code, hmobstore was modified to traverse an hfile link to a mob. Ideally this should use the transparent filelink code to read the data. Also there will likely be some issues with the mob file cache with these links. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205827#comment-14205827
]
Hadoop QA commented on HBASE-12454:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680706/HBASE-12454.patch
against trunk revision .
ATTACHMENT ID: 12680706
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11629//console
This message is automatically generated.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12454.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if
[jira] [Commented] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[ https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205828#comment-14205828 ] Qiang Tian commented on HBASE-12451: Minimum split size is good and simple enough for me..user could have a tradeoff between automatic tuning and customization based on knowledge of their workload...(many times we do not want to expose too many configure parameters, but it looks really useful in some cases:-)) based on total region count looks hard to control, e.g. if user pre-split many regions, e.g. in http://search-hadoop.com/m/DHED4aS08G1 with 240 regions, the size will be quite big unless hbase.increasing.policy.initial.size is also configured.. IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster Key: HBASE-12451 URL: https://issues.apache.org/jira/browse/HBASE-12451 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minimum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[
https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205866#comment-14205866
]
zhangduo commented on HBASE-12451:
--
I think there is a upper limit also, so the size will not grow too large...
{code}
protected long getSizeToCheck(final int tableRegionsCount) {
// safety check for 100 to avoid numerical overflow in extreme cases
return tableRegionsCount == 0 || tableRegionsCount 100 ?
getDesiredMaxFileSize():
Math.min(getDesiredMaxFileSize(),
this.initialSize * tableRegionsCount * tableRegionsCount *
tableRegionsCount);
}
this.desiredMaxFileSize = conf.getLong(HConstants.HREGION_MAX_FILESIZE,
HConstants.DEFAULT_MAX_FILE_SIZE);
/** Conf key for the max file size after which we split the region */
public static final String HREGION_MAX_FILESIZE =
hbase.hregion.max.filesize;
/** Default maximum file size */
public static final long DEFAULT_MAX_FILE_SIZE = 10 * 1024 * 1024 * 1024L;
{code}
We have two reason to split, first is for load balancing, this introduces a
increasing region split size, and second is for compaction, this introduces a
constant region split size(which is the upper limit).
I think the first thing need to be done is define unnecessary region splits.
If we already have 240 regions of a table, and there is only one region of this
table on a regionserver, should the region have a small split size?
IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits
in rolling update of cluster
Key: HBASE-12451
URL: https://issues.apache.org/jira/browse/HBASE-12451
Project: HBase
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0
Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split
policy. In this policy, split size is the number of regions that are on this
server that all are of the same table, cubed, times 2x the region flush size.
But when unloading regions of a regionserver in a cluster using
region_mover.rb, the number of regions that are on this server that all are
of the same table will decrease, and the split size will decrease too, which
may cause the left region split in the regionsever. Region Splits also
happens when loading regions of a regionserver in a cluster.
A improvment may set a minimum split size in
IncreasingToUpperBoundRegionSplitPolicy
Suggestions are welcomed. Thanks~
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-10773) Make use of ByteRanges in HFileBlock instead of ByteBuffers
[ https://issues.apache.org/jira/browse/HBASE-10773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10773: --- Attachment: HBASE-10772_2.patch Just parking an older patch for referernce. Tries to bring PBR in the BlockCache also. Make use of ByteRanges in HFileBlock instead of ByteBuffers --- Key: HBASE-10773 URL: https://issues.apache.org/jira/browse/HBASE-10773 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-10772_2.patch, HBASE-10772_3.patch Replacing BBs with Byte Ranges in block cache as part of HBASE-10772, would help in replacing BBs with BRs in HFileBlock also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12444) Total number of requests overflow because it's int
[
https://issues.apache.org/jira/browse/HBASE-12444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205900#comment-14205900
]
Hadoop QA commented on HBASE-12444:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680722/hbase-12444-v1.patch
against trunk revision .
ATTACHMENT ID: 12680722
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11630//console
This message is automatically generated.
Total number of requests overflow because it's int
--
Key: HBASE-12444
URL: https://issues.apache.org/jira/browse/HBASE-12444
Project: HBase
Issue Type: Bug
Components: hbck, master, regionserver
Reporter: zhaoyunjiong
Priority: Minor
Attachments: hbase-12444-v1.patch, hbase-12444.patch
When running hbck, I noticed Number of requests was wrong:
Average load: 466.41237113402065
Number of requests: -1835941345
Number of regions: 45242
Number of regions in transition: 0
The root cause is it use int, and clearly it overflowed.
I'll update a patch later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12433) Coprocessors not dynamically reordered when reset priority
[ https://issues.apache.org/jira/browse/HBASE-12433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205951#comment-14205951 ] Jingcheng Du commented on HBASE-12433: -- Hi James [~jamestaylor], you changed the coprocessor priority by altering table in HBase shell? Did you disabled/enabled the table at that time? Coprocessors not dynamically reordered when reset priority -- Key: HBASE-12433 URL: https://issues.apache.org/jira/browse/HBASE-12433 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.7 Reporter: James Taylor When modifying the coprocessor priority through the HBase shell, the order of the firing of the coprocessors wasn't changing. It probably would have with a cluster bounce, but if we can make it dynamic easily, that would be preferable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12433) Coprocessors not dynamically reordered when reset priority
[ https://issues.apache.org/jira/browse/HBASE-12433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205967#comment-14205967 ] James Taylor commented on HBASE-12433: -- No, no disable/enable. I thought that wasn't necessary anymore? Coprocessors not dynamically reordered when reset priority -- Key: HBASE-12433 URL: https://issues.apache.org/jira/browse/HBASE-12433 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.7 Reporter: James Taylor When modifying the coprocessor priority through the HBase shell, the order of the firing of the coprocessors wasn't changing. It probably would have with a cluster bounce, but if we can make it dynamic easily, that would be preferable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12455) Add 'description' to bean and attribute output when you do /jmx?description=true
stack created HBASE-12455: - Summary: Add 'description' to bean and attribute output when you do /jmx?description=true Key: HBASE-12455 URL: https://issues.apache.org/jira/browse/HBASE-12455 Project: HBase Issue Type: Improvement Components: metrics Reporter: stack Its hard figuring what our metrics mean. Each attribute and bean actually has a description but its hard to get at. In mission control, etc., you have to click on each attribute to see the description. Its painful. Because the description is rarely read, they are not as info-full as they could be. If you do /jmx in the UI, you get a dump of all beans associated with the server but its just the attribute/bean name + value. The description is there but its not displayed. We should give option to display descriptions. It would be good for those exploring what metrics are available. We actually point folks at jvisualvm in the refguide to figure what metrics are available. Would be useful if we could point them at something that was easier to navigate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12455) Add 'description' to bean and attribute output when you do /jmx?description=true
[
https://issues.apache.org/jira/browse/HBASE-12455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12455:
--
Attachment: 12455.txt
I tried to make it so if you added a '?description=true' param on your /jmx
query, then we'd print out descriptions. Unfortunately it will take more work
than I've done here.
Here is it working:
{code}
...
name : regionCount,
description : Number of regions,
value : 2,
name : storeCount,
description : Number of Stores,
value : 2,
name : hlogFileCount,
description : Number of HLog Files,
value : 2,
name : hlogFileSize,
description : Size of all HLog Files,
value : 0,
name : storeFileCount,
description : Number of Store Files,
value : 2,
name : memStoreSize,
description : Size of the memstore,
value : 26880,
name : storeFileSize,
description : Size of storefiles being served.,
value : 14415,
name : regionServerStartTime,
description : RegionServer Start Time,
value : 1415683550907,
name : totalRequestCount,
description : Total number of requests this RegionServer has answered.,
value : 53,
name : readRequestCount,
description : Number of read requests this region server has answered.,
value : 13,
name : writeRequestCount,
description : Number of mutation requests this region server has
answered.,
value : 42,
name : checkMutateFailedCount,
description : Number of Check and Mutate calls that failed the checks.,
value : 0,
name : checkMutatePassedCount,
description : Number of Check and Mutate calls that passed the checks.,
value : 0,
name : storeFileIndexSize,
description : Size of indexes in storefiles on disk.,
value : 848,
...
{code}
But it then goes off the rails when beans and/or attributes are without
description:
{code}
...
name : java.lang:type=MemoryPool,name=CMS Old Gen,
description : Information on the management interface of the MBean,
modelerType : sun.management.MemoryPoolImpl,
name : Name,
description : Name,
value : CMS Old Gen,
name : Type,
description : Type,
value : HEAP,
name : Valid,
description : Valid,
value : true,
name : Usage,
description : Usage,
value : {
},
name : PeakUsage,
description : PeakUsage,
value : {
},
name : MemoryManagerNames,
description : MemoryManagerNames,
value : [ ConcurrentMarkSweep ],
name : UsageThreshold,
description : UsageThreshold,
value : 0,
...
{code}
Add 'description' to bean and attribute output when you do
/jmx?description=true
Key: HBASE-12455
URL: https://issues.apache.org/jira/browse/HBASE-12455
Project: HBase
Issue Type: Improvement
Components: metrics
Reporter: stack
Attachments: 12455.txt
Its hard figuring what our metrics mean. Each attribute and bean actually has
a description but its hard to get at. In mission control, etc., you have to
click on each attribute to see the description. Its painful. Because the
description is rarely read, they are not as info-full as they could be.
If you do /jmx in the UI, you get a dump of all beans associated with the
server but its just the attribute/bean name + value. The description is there
but its not displayed. We should give option to display descriptions. It
would be good for those exploring what metrics are available. We actually
point folks at jvisualvm in the refguide to figure what metrics are
available. Would be useful if we could point them at something that was
easier to navigate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12455) Add 'description' to bean and attribute output when you do /jmx?description=true
[ https://issues.apache.org/jira/browse/HBASE-12455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205991#comment-14205991 ] stack commented on HBASE-12455: --- Something like this might be OTT https://code.google.com/p/purej-vminspect/ Add 'description' to bean and attribute output when you do /jmx?description=true Key: HBASE-12455 URL: https://issues.apache.org/jira/browse/HBASE-12455 Project: HBase Issue Type: Improvement Components: metrics Reporter: stack Attachments: 12455.txt Its hard figuring what our metrics mean. Each attribute and bean actually has a description but its hard to get at. In mission control, etc., you have to click on each attribute to see the description. Its painful. Because the description is rarely read, they are not as info-full as they could be. If you do /jmx in the UI, you get a dump of all beans associated with the server but its just the attribute/bean name + value. The description is there but its not displayed. We should give option to display descriptions. It would be good for those exploring what metrics are available. We actually point folks at jvisualvm in the refguide to figure what metrics are available. Would be useful if we could point them at something that was easier to navigate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12456) Update surefire from 2.18-SNAPSHOT to 2.18
stack created HBASE-12456: - Summary: Update surefire from 2.18-SNAPSHOT to 2.18 Key: HBASE-12456 URL: https://issues.apache.org/jira/browse/HBASE-12456 Project: HBase Issue Type: Task Components: test Reporter: stack Assignee: stack ...before SNAPSHOT evaporates -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12456) Update surefire from 2.18-SNAPSHOT to 2.18
[ https://issues.apache.org/jira/browse/HBASE-12456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12456: -- Attachment: 12456.txt Simple patch. Update surefire from 2.18-SNAPSHOT to 2.18 -- Key: HBASE-12456 URL: https://issues.apache.org/jira/browse/HBASE-12456 Project: HBase Issue Type: Task Components: test Reporter: stack Assignee: stack Attachments: 12456.txt ...before SNAPSHOT evaporates -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12456) Update surefire from 2.18-SNAPSHOT to 2.18
[ https://issues.apache.org/jira/browse/HBASE-12456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-12456. --- Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Pushed to branch-1+ Update surefire from 2.18-SNAPSHOT to 2.18 -- Key: HBASE-12456 URL: https://issues.apache.org/jira/browse/HBASE-12456 Project: HBase Issue Type: Task Components: test Reporter: stack Assignee: stack Fix For: 2.0.0, 0.99.2 Attachments: 12456.txt ...before SNAPSHOT evaporates -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12457) Regions in transition for a long time when CLOSE interleaves with a slow compaction
Lars Hofhansl created HBASE-12457: - Summary: Regions in transition for a long time when CLOSE interleaves with a slow compaction Key: HBASE-12457 URL: https://issues.apache.org/jira/browse/HBASE-12457 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Under heave load we have observed regions remaining in transition for 20 minutes when the master requests a close while a slow compaction is running. The pattern is always something like this: # RS starts a compaction # HM request the region to be closed on this RS # Compaction is not aborted for another 20 minutes # The region is in transition and not usable. In every case I tracked down so far the time between the requested CLOSE and abort of the compaction is almost exactly 20 minutes, which is suspicious. Of course part of the issue is having compactions that take over 20 minutes, but maybe we can do better here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12457) Regions in transition for a long time when CLOSE interleaves with a slow compaction
[ https://issues.apache.org/jira/browse/HBASE-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-12457: -- Affects Version/s: 0.98.7 Regions in transition for a long time when CLOSE interleaves with a slow compaction --- Key: HBASE-12457 URL: https://issues.apache.org/jira/browse/HBASE-12457 Project: HBase Issue Type: Bug Affects Versions: 0.98.7 Reporter: Lars Hofhansl Under heave load we have observed regions remaining in transition for 20 minutes when the master requests a close while a slow compaction is running. The pattern is always something like this: # RS starts a compaction # HM request the region to be closed on this RS # Compaction is not aborted for another 20 minutes # The region is in transition and not usable. In every case I tracked down so far the time between the requested CLOSE and abort of the compaction is almost exactly 20 minutes, which is suspicious. Of course part of the issue is having compactions that take over 20 minutes, but maybe we can do better here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12457) Regions in transition for a long time when CLOSE interleaves with a slow compaction
[ https://issues.apache.org/jira/browse/HBASE-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206010#comment-14206010 ] Lars Hofhansl commented on HBASE-12457: --- Sometime (but not always) we Splits interspersed with this. While l scoured over the code I noticed the following: * SplitTransaction write CREATE_SPLIT_DIR after it created the daugther dirs and CLOSED_PARENT_REGION after the parent region is closed * Upon rollback writestate.writesEnabled is set back to true unconditionally at the CREATE_SPLIT_DIR stage. It seems that should only be done when we journaled CLOSED_PARENT_REGION. Regions in transition for a long time when CLOSE interleaves with a slow compaction --- Key: HBASE-12457 URL: https://issues.apache.org/jira/browse/HBASE-12457 Project: HBase Issue Type: Bug Affects Versions: 0.98.7 Reporter: Lars Hofhansl Under heave load we have observed regions remaining in transition for 20 minutes when the master requests a close while a slow compaction is running. The pattern is always something like this: # RS starts a compaction # HM request the region to be closed on this RS # Compaction is not aborted for another 20 minutes # The region is in transition and not usable. In every case I tracked down so far the time between the requested CLOSE and abort of the compaction is almost exactly 20 minutes, which is suspicious. Of course part of the issue is having compactions that take over 20 minutes, but maybe we can do better here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12457) Regions in transition for a long time when CLOSE interleaves with a slow compaction
[ https://issues.apache.org/jira/browse/HBASE-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-12457: -- Attachment: 12457-minifix.txt Here's a minifix for this issue. Note: This is *not* the core of the problem, just something I noticed. Regions in transition for a long time when CLOSE interleaves with a slow compaction --- Key: HBASE-12457 URL: https://issues.apache.org/jira/browse/HBASE-12457 Project: HBase Issue Type: Bug Affects Versions: 0.98.7 Reporter: Lars Hofhansl Attachments: 12457-minifix.txt Under heave load we have observed regions remaining in transition for 20 minutes when the master requests a close while a slow compaction is running. The pattern is always something like this: # RS starts a compaction # HM request the region to be closed on this RS # Compaction is not aborted for another 20 minutes # The region is in transition and not usable. In every case I tracked down so far the time between the requested CLOSE and abort of the compaction is almost exactly 20 minutes, which is suspicious. Of course part of the issue is having compactions that take over 20 minutes, but maybe we can do better here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206039#comment-14206039 ] Stephen Yuan Jiang commented on HBASE-6028: --- [~esteban] are you actively working on this? If not, I can take over the investigation. Implement a cancel for in-progress compactions -- Key: HBASE-6028 URL: https://issues.apache.org/jira/browse/HBASE-6028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Derek Wollenstein Assignee: Esteban Gutierrez Priority: Minor Labels: beginner Depending on current server load, it can be extremely expensive to run periodic minor / major compactions. It would be helpful to have a feature where a user could use the shell or a client tool to explicitly cancel an in-progress compactions. This would allow a system to recover when too many regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[ https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206040#comment-14206040 ] Qiang Tian commented on HBASE-12451: Thanks Duo, forgot that. with default values, it looks the region split size will use upper limit after 3 regions. according to http://hbase.apache.org/book/ops.capacity.html, region count and region size are most important factors, but there is no clear answer for region count? bq. If we already have 240 regions of a table, and there is only one region of this table on a regionserver, should the region have a small split size? the regions should be evenly spread across RS(8 RS in that case) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster Key: HBASE-12451 URL: https://issues.apache.org/jira/browse/HBASE-12451 Project: HBase Issue Type: Bug Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split policy. In this policy, split size is the number of regions that are on this server that all are of the same table, cubed, times 2x the region flush size. But when unloading regions of a regionserver in a cluster using region_mover.rb, the number of regions that are on this server that all are of the same table will decrease, and the split size will decrease too, which may cause the left region split in the regionsever. Region Splits also happens when loading regions of a regionserver in a cluster. A improvment may set a minimum split size in IncreasingToUpperBoundRegionSplitPolicy Suggestions are welcomed. Thanks~ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206048#comment-14206048 ] Esteban Gutierrez commented on HBASE-6028: -- Hi [~syuanjiang], yes I'm working on this I have a prototype for trunk and only needs some clean up and expose this functionality via the hbase shell. Implement a cancel for in-progress compactions -- Key: HBASE-6028 URL: https://issues.apache.org/jira/browse/HBASE-6028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Derek Wollenstein Assignee: Esteban Gutierrez Priority: Minor Labels: beginner Depending on current server load, it can be extremely expensive to run periodic minor / major compactions. It would be helpful to have a feature where a user could use the shell or a client tool to explicitly cancel an in-progress compactions. This would allow a system to recover when too many regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206054#comment-14206054
]
stack commented on HBASE-12454:
---
Looks good Andrew.
Could runner.compaction go null between check and usage on #560?
559 if (runner.compaction != null) {
560 runner.store.cancelRequestedCompaction(runner.compaction);
561 }
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12454.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12457) Regions in transition for a long time when CLOSE interleaves with a slow compaction
[ https://issues.apache.org/jira/browse/HBASE-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206059#comment-14206059 ] Lars Hofhansl commented on HBASE-12457: --- That all said, in the end I have observed that only on a single region server (so far) so might be an environmental issue. Regions in transition for a long time when CLOSE interleaves with a slow compaction --- Key: HBASE-12457 URL: https://issues.apache.org/jira/browse/HBASE-12457 Project: HBase Issue Type: Bug Affects Versions: 0.98.7 Reporter: Lars Hofhansl Attachments: 12457-minifix.txt Under heave load we have observed regions remaining in transition for 20 minutes when the master requests a close while a slow compaction is running. The pattern is always something like this: # RS starts a compaction # HM request the region to be closed on this RS # Compaction is not aborted for another 20 minutes # The region is in transition and not usable. In every case I tracked down so far the time between the requested CLOSE and abort of the compaction is almost exactly 20 minutes, which is suspicious. Of course part of the issue is having compactions that take over 20 minutes, but maybe we can do better here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12454) Setting didPerformCompaction early in HRegion#compact
[
https://issues.apache.org/jira/browse/HBASE-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206062#comment-14206062
]
Andrew Purtell commented on HBASE-12454:
I can move the null check closer to where it is used.
Setting didPerformCompaction early in HRegion#compact
-
Key: HBASE-12454
URL: https://issues.apache.org/jira/browse/HBASE-12454
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.8
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: HBASE-12454.patch, HBASE-12454.patch
It appears we are setting 'didPerformCompaction' to true before attempting
the compaction in HRegion#compact. If Store#compact throws an exception or is
interrupted, we won't call Store#cancelRequestedCompaction in the last
finally block of the method as it looks like we should.
{code}
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -1540,58 +1540,58 @@ public class HRegion implements HeapSize,
PropagatingConfigurationObserver {
return false;
}
status = TaskMonitor.get().createStatus(Compacting + store + in
+ this);
if (this.closed.get()) {
String msg = Skipping compaction on + this + because closed;
LOG.debug(msg);
status.abort(msg);
return false;
}
boolean wasStateSet = false;
try {
synchronized (writestate) {
if (writestate.writesEnabled) {
wasStateSet = true;
++writestate.compacting;
} else {
String msg = NOT compacting region + this + . Writes
disabled.;
LOG.info(msg);
status.abort(msg);
return false;
}
}
LOG.info(Starting compaction on + store + in region + this
+ (compaction.getRequest().isOffPeak()? as an off-peak
compaction:));
doRegionCompactionPrep();
try {
status.setStatus(Compacting store + store);
- didPerformCompaction = true;
store.compact(compaction);
+ didPerformCompaction = true;
} catch (InterruptedIOException iioe) {
String msg = compaction interrupted;
LOG.info(msg, iioe);
status.abort(msg);
return false;
}
} finally {
if (wasStateSet) {
synchronized (writestate) {
--writestate.compacting;
if (writestate.compacting = 0) {
writestate.notifyAll();
}
}
}
}
status.markComplete(Compaction complete);
return true;
} finally {
try {
if (!didPerformCompaction)
store.cancelRequestedCompaction(compaction); -
if (status != null) status.cleanup();
} finally {
lock.readLock().unlock();
}
}
}{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12456) Update surefire from 2.18-SNAPSHOT to 2.18
[ https://issues.apache.org/jira/browse/HBASE-12456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206066#comment-14206066 ] Hudson commented on HBASE-12456: SUCCESS: Integrated in HBase-TRUNK #5758 (See [https://builds.apache.org/job/HBase-TRUNK/5758/]) HBASE-12456 Update surefire from 2.18-SNAPSHOT to 2.18 (stack: rev df8859d5a55d02c20c8f4234f65c97c7acfcaa7e) * pom.xml Update surefire from 2.18-SNAPSHOT to 2.18 -- Key: HBASE-12456 URL: https://issues.apache.org/jira/browse/HBASE-12456 Project: HBase Issue Type: Task Components: test Reporter: stack Assignee: stack Fix For: 2.0.0, 0.99.2 Attachments: 12456.txt ...before SNAPSHOT evaporates -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12451) IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits in rolling update of cluster
[
https://issues.apache.org/jira/browse/HBASE-12451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206072#comment-14206072
]
zhangduo commented on HBASE-12451:
--
{quote}
the regions should be evenly spread across RS(8 RS in that case)
{quote}
But there are exceptions, see the description of this issue...
And I made a mistake above. Just use total region count of table is not enough,
we also need the total regionserver count as a parameter. 240 regions is not
necessary for 8 regionservers, but is OK for 80 regionservers, right?
IncreasingToUpperBoundRegionSplitPolicy may cause unnecessary region splits
in rolling update of cluster
Key: HBASE-12451
URL: https://issues.apache.org/jira/browse/HBASE-12451
Project: HBase
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0
Currently IncreasingToUpperBoundRegionSplitPolicy is the default region split
policy. In this policy, split size is the number of regions that are on this
server that all are of the same table, cubed, times 2x the region flush size.
But when unloading regions of a regionserver in a cluster using
region_mover.rb, the number of regions that are on this server that all are
of the same table will decrease, and the split size will decrease too, which
may cause the left region split in the regionsever. Region Splits also
happens when loading regions of a regionserver in a cluster.
A improvment may set a minimum split size in
IncreasingToUpperBoundRegionSplitPolicy
Suggestions are welcomed. Thanks~
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12433) Coprocessors not dynamically reordered when reset priority
[ https://issues.apache.org/jira/browse/HBASE-12433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206094#comment-14206094 ] Jingcheng Du commented on HBASE-12433: -- No, there wasn't. The coprocessors could be reloaded/reordered when the regions are instantiated. It means if the coprocessor priority is changed, disabling/enabling the table would help reorder the coprocessors in regions. Is that possible to disable/enable tables in your case? Thanks! Coprocessors not dynamically reordered when reset priority -- Key: HBASE-12433 URL: https://issues.apache.org/jira/browse/HBASE-12433 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.7 Reporter: James Taylor When modifying the coprocessor priority through the HBase shell, the order of the firing of the coprocessors wasn't changing. It probably would have with a cluster bounce, but if we can make it dynamic easily, that would be preferable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12456) Update surefire from 2.18-SNAPSHOT to 2.18
[ https://issues.apache.org/jira/browse/HBASE-12456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206096#comment-14206096 ] Hudson commented on HBASE-12456: SUCCESS: Integrated in HBase-1.0 #449 (See [https://builds.apache.org/job/HBase-1.0/449/]) HBASE-12456 Update surefire from 2.18-SNAPSHOT to 2.18 (stack: rev c2a17ad6ea0dbd2bcbaa538e8db81a665053f58f) * pom.xml Update surefire from 2.18-SNAPSHOT to 2.18 -- Key: HBASE-12456 URL: https://issues.apache.org/jira/browse/HBASE-12456 Project: HBase Issue Type: Task Components: test Reporter: stack Assignee: stack Fix For: 2.0.0, 0.99.2 Attachments: 12456.txt ...before SNAPSHOT evaporates -- This message was sent by Atlassian JIRA (v6.3.4#6332)
