[jira] [Commented] (HBASE-9719) Premptive Call Me Maybe HBase
[ https://issues.apache.org/jira/browse/HBASE-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793935#comment-13793935 ] Lars Hofhansl commented on HBASE-9719: -- Should not be a problem in HBase, since a single RegionServer is wholly responsible for a row at any given time. Depending on which problem we're talking about there could still be a problem if RegionServers times are further apart than it takes a region to be migrated to another RegionServer. Then there is the question about events that do happen at the same time (or within the time resolution, which is 1ms in HBase). Generally HBase has a pretty good story here. Premptive Call Me Maybe HBase - Key: HBASE-9719 URL: https://issues.apache.org/jira/browse/HBASE-9719 Project: HBase Issue Type: Bug Reporter: stack Aphyr wrote an interesting article on C* [1]. Some awkward-looking issues were turned up though it seems the author is purportedly doing nothing but exercising the software within spec; he is just paying close attention to what is being returned. It does not look like Aphyr will be coming our way any time soon [2] -- thanks Ian Varley -- but he could change his mind. Wouldn't it be coolio if we'd already run his test suite and found any bugs and fixed them before he came by? This issue is about running his article against hbase so we find the embarrassing before he does. 1. http://aphyr.com/posts/294-call-me-maybe-cassandra 2. https://twitter.com/aphyr/status/335082835868254209 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9719) Premptive Call Me Maybe HBase
[ https://issues.apache.org/jira/browse/HBASE-9719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793938#comment-13793938 ] Cosmin Lehene commented on HBASE-9719: -- [~lhofhansl] I saw your slides about MVCC writenumber (http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final) There are, however, a few scenarios related to the wall clock that we should validate. There's also the replication. Premptive Call Me Maybe HBase - Key: HBASE-9719 URL: https://issues.apache.org/jira/browse/HBASE-9719 Project: HBase Issue Type: Bug Reporter: stack Aphyr wrote an interesting article on C* [1]. Some awkward-looking issues were turned up though it seems the author is purportedly doing nothing but exercising the software within spec; he is just paying close attention to what is being returned. It does not look like Aphyr will be coming our way any time soon [2] -- thanks Ian Varley -- but he could change his mind. Wouldn't it be coolio if we'd already run his test suite and found any bugs and fixed them before he came by? This issue is about running his article against hbase so we find the embarrassing before he does. 1. http://aphyr.com/posts/294-call-me-maybe-cassandra 2. https://twitter.com/aphyr/status/335082835868254209 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9751) Excessive readpoints checks in StoreFileScanner
[ https://issues.apache.org/jira/browse/HBASE-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793943#comment-13793943 ] Hudson commented on HBASE-9751: --- SUCCESS: Integrated in HBase-TRUNK #4613 (See [https://builds.apache.org/job/HBase-TRUNK/4613/]) HBASE-9751 Excessive readpoints checks in StoreFileScanner (larsh: rev 1531790) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java Excessive readpoints checks in StoreFileScanner Key: HBASE-9751 URL: https://issues.apache.org/jira/browse/HBASE-9751 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.94.12, 0.96.0 Reporter: Vladimir Rodionov Assignee: Lars Hofhansl Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: 9751-0.94.txt, 9751-0.94-v2.txt, 9751-trunk.txt, 9751-trunk-v2.txt It seems that usage of skipKVsNewerThanReadpoint in StoreFileScanner can be greatly reduced or even eliminated all together (HFiles are immutable and no new KVs can be inserted after scanner instance is created). The same is true for MemStoreScanner which checks readpoint on every next() and seek(). Each readpoint check is ThreadLocal.get() and it is quite expensive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9751) Excessive readpoints checks in StoreFileScanner
[ https://issues.apache.org/jira/browse/HBASE-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793948#comment-13793948 ] Hudson commented on HBASE-9751: --- SUCCESS: Integrated in hbase-0.96 #137 (See [https://builds.apache.org/job/hbase-0.96/137/]) HBASE-9751 Excessive readpoints checks in StoreFileScanner (larsh: rev 1531791) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java Excessive readpoints checks in StoreFileScanner Key: HBASE-9751 URL: https://issues.apache.org/jira/browse/HBASE-9751 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.94.12, 0.96.0 Reporter: Vladimir Rodionov Assignee: Lars Hofhansl Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: 9751-0.94.txt, 9751-0.94-v2.txt, 9751-trunk.txt, 9751-trunk-v2.txt It seems that usage of skipKVsNewerThanReadpoint in StoreFileScanner can be greatly reduced or even eliminated all together (HFiles are immutable and no new KVs can be inserted after scanner instance is created). The same is true for MemStoreScanner which checks readpoint on every next() and seek(). Each readpoint check is ThreadLocal.get() and it is quite expensive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Menghani reassigned HBASE-3149: -- Assignee: Gaurav Menghani Assigning this to myself, since I have committed this change in the 0.89-fb branch. Will upload the patch shortly. Make flush decisions per column family -- Key: HBASE-3149 URL: https://issues.apache.org/jira/browse/HBASE-3149 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Karthik Ranganathan Assignee: Gaurav Menghani Priority: Critical Today, the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-3727) MultiHFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793980#comment-13793980 ] Viral Bajaria commented on HBASE-3727: -- +1 to what James says here. We also have a need for something similar. Is it possible to include this patch in next release ? Or are there any concerns ? MultiHFileOutputFormat -- Key: HBASE-3727 URL: https://issues.apache.org/jira/browse/HBASE-3727 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Priority: Minor Attachments: MultiHFileOutputFormat.java, MultiHFileOutputFormat.java Like MultiTableOutputFormat, but outputting HFiles. Key is tablename as an IBW. Creates sub-writers (code cut and pasted from HFileOutputFormat) on demand that produce HFiles in per-table subdirectories of the configured output path. Does not currently support partitioning for existing tables / incremental update. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9605) Allow AggregationClient to skip specifying column family for row count aggregate
[
https://issues.apache.org/jira/browse/HBASE-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chendihao updated HBASE-9605:
-
Attachment: 0605-0.94.patch
patch for 0.94
Allow AggregationClient to skip specifying column family for row count
aggregate
Key: HBASE-9605
URL: https://issues.apache.org/jira/browse/HBASE-9605
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 0605-0.94.patch, 9605-v1.txt
For rowcounter job, column family is not required as input parameter.
AggregationClient requires the specification of one column family:
{code}
} else if (scan.getFamilyMap().size() != 1) {
throw new IOException(There must be only one family.);
}
{code}
We should relax the above requirement for row count aggregate where
FirstKeyOnlyFilter would be automatically applied.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9605) Allow AggregationClient to skip specifying column family for row count aggregate
[
https://issues.apache.org/jira/browse/HBASE-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794016#comment-13794016
]
Hadoop QA commented on HBASE-9605:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608258/0605-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7536//console
This message is automatically generated.
Allow AggregationClient to skip specifying column family for row count
aggregate
Key: HBASE-9605
URL: https://issues.apache.org/jira/browse/HBASE-9605
Project: HBase
Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 0605-0.94.patch, 9605-v1.txt
For rowcounter job, column family is not required as input parameter.
AggregationClient requires the specification of one column family:
{code}
} else if (scan.getFamilyMap().size() != 1) {
throw new IOException(There must be only one family.);
}
{code}
We should relax the above requirement for row count aggregate where
FirstKeyOnlyFilter would be automatically applied.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9751) Excessive readpoints checks in StoreFileScanner
[ https://issues.apache.org/jira/browse/HBASE-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794078#comment-13794078 ] Hudson commented on HBASE-9751: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #87 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/87/]) HBASE-9751 Excessive readpoints checks in StoreFileScanner (larsh: rev 1531791) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java Excessive readpoints checks in StoreFileScanner Key: HBASE-9751 URL: https://issues.apache.org/jira/browse/HBASE-9751 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.94.12, 0.96.0 Reporter: Vladimir Rodionov Assignee: Lars Hofhansl Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: 9751-0.94.txt, 9751-0.94-v2.txt, 9751-trunk.txt, 9751-trunk-v2.txt It seems that usage of skipKVsNewerThanReadpoint in StoreFileScanner can be greatly reduced or even eliminated all together (HFiles are immutable and no new KVs can be inserted after scanner instance is created). The same is true for MemStoreScanner which checks readpoint on every next() and seek(). Each readpoint check is ThreadLocal.get() and it is quite expensive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9751) Excessive readpoints checks in StoreFileScanner
[ https://issues.apache.org/jira/browse/HBASE-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794093#comment-13794093 ] Hudson commented on HBASE-9751: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #792 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/792/]) HBASE-9751 Excessive readpoints checks in StoreFileScanner (larsh: rev 1531790) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java Excessive readpoints checks in StoreFileScanner Key: HBASE-9751 URL: https://issues.apache.org/jira/browse/HBASE-9751 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.94.12, 0.96.0 Reporter: Vladimir Rodionov Assignee: Lars Hofhansl Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: 9751-0.94.txt, 9751-0.94-v2.txt, 9751-trunk.txt, 9751-trunk-v2.txt It seems that usage of skipKVsNewerThanReadpoint in StoreFileScanner can be greatly reduced or even eliminated all together (HFiles are immutable and no new KVs can be inserted after scanner instance is created). The same is true for MemStoreScanner which checks readpoint on every next() and seek(). Each readpoint check is ThreadLocal.get() and it is quite expensive. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9747) PrefixFilter with OR condition gives wrong results
[
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794131#comment-13794131
]
Deepa Remesh commented on HBASE-9747:
-
The patch works for me too. Thanks.
PrefixFilter with OR condition gives wrong results
--
Key: HBASE-9747
URL: https://issues.apache.org/jira/browse/HBASE-9747
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch
PrefixFilter when used with a SingleColumnValueFilter with an OR condition
gives wrong results. In below example, each filter when evaluated separately
gives 1 row each. The OR condition with the two filters gives 3 rows instead
of 2. Repro below:
create 't1', 'f1'
put 't1','a1','f1:q2','111'
put 't1','b1','f1:q1','112'
put 't1','c1','f1:q1','113'
hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR
SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
ROWCOLUMN+CELL
a1column=f1:q2,
timestamp=1381468905492, value=111
b1column=f1:q1,
timestamp=1381468905518, value=112
c1column=f1:q1,
timestamp=1381468905549, value=113
3 row(s) in 0.1020 seconds
hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
ROWCOLUMN+CELL
b1column=f1:q1,
timestamp=1381468905518, value=112
1 row(s) in 0.0150 seconds
hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1',
=, 'binary:113')}
ROWCOLUMN+CELL
c1column=f1:q1,
timestamp=1381469178679, value=113
1 row(s) in 0.0140 seconds
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9137) Add Tag dictionary in WAL compression
[ https://issues.apache.org/jira/browse/HBASE-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794140#comment-13794140 ] Ted Yu commented on HBASE-9137: --- lgtm. Add Tag dictionary in WAL compression - Key: HBASE-9137 URL: https://issues.apache.org/jira/browse/HBASE-9137 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9137.patch, HBASE-9137_V2.patch, HBASE-9137_V3.patch We can add tag dictionary like we have one for rowdictionary, familydictionary. But this has to be done after stabilizing HBASE-7391. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794232#comment-13794232 ] Jimmy Xiang commented on HBASE-5487: Good. I think we are on the same page. bq. just using it as a reliable storage. We probably won't use ZK as a pure storage. Meta table + cache is a good alternative. Generic framework for Master-coordinated tasks -- Key: HBASE-5487 URL: https://issues.apache.org/jira/browse/HBASE-5487 Project: HBase Issue Type: New Feature Components: master, regionserver, Zookeeper Affects Versions: 0.94.0 Reporter: Mubarak Seyed Priority: Critical Attachments: Region management in Master.pdf Need a framework to execute master-coordinated tasks in a fault-tolerant manner. Master-coordinated tasks such as online-scheme change and delete-range (deleting region(s) based on start/end key) can make use of this framework. The advantages of framework are 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for master-coordinated tasks 2. Ability to abstract the common functions across Master - ZK and RS - ZK 3. Easy to plugin new master-coordinated tasks without adding code to core components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794254#comment-13794254 ] stack commented on HBASE-3149: -- Thanks [~gaurav.menghani] Make flush decisions per column family -- Key: HBASE-3149 URL: https://issues.apache.org/jira/browse/HBASE-3149 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Karthik Ranganathan Assignee: Gaurav Menghani Priority: Critical Today, the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
Jimmy Xiang created HBASE-9757: -- Summary: Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
[ https://issues.apache.org/jira/browse/HBASE-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794259#comment-13794259 ] Jimmy Xiang commented on HBASE-9757: With 0.96 tip, after fast region move re-enabled, ITBLL is green for me. However, there seems to be some issue with trunk. Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9758) Log missing rows and their regions if ITBLL fails
Jimmy Xiang created HBASE-9758: -- Summary: Log missing rows and their regions if ITBLL fails Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9758: --- Attachment: trunk-9758.patch Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9757) Reenable fast region move in SlowDeterministicMonkey
[ https://issues.apache.org/jira/browse/HBASE-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794264#comment-13794264 ] stack commented on HBASE-9757: -- I'd say lets figure the issues we have first before we up the ante. After we figure whats up and then a couple of greens, lets up the pressure. We should reenable your fast move as you suggest here. Should we then try reenabling online-config tests as you may have fixed what was up w/ it? Then we should move to kills that do in the RS+DN, not just RS (As Elliott suggested Friday). That'll be a new set of issues for sure. Good on you Jimmy. Reenable fast region move in SlowDeterministicMonkey Key: HBASE-9757 URL: https://issues.apache.org/jira/browse/HBASE-9757 Project: HBase Issue Type: Task Components: test Reporter: Jimmy Xiang Assignee: Jimmy Xiang HBASE-9338 slows down the region move CM a little so that ITBLL is green for 0.96.0 RC. We should revert the change and make sure the test is still green. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794269#comment-13794269 ] stack commented on HBASE-9758: -- +1 on patch for trunk and 0.96. I suppose the region could have changed since the fail but this is way better than what we currently have. Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9758: --- Status: Patch Available (was: Open) This patch logs at most 50 missing rows and the corresponding regions that hold these rows. Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7379) Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk
[ https://issues.apache.org/jira/browse/HBASE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794279#comment-13794279 ] Ted Yu commented on HBASE-7379: --- Looks like this is no longer needed. Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk --- Key: HBASE-7379 URL: https://issues.apache.org/jira/browse/HBASE-7379 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Priority: Minor Attachments: 7379-trunk.txt, 7379-trunk-v2.txt HBASE-6728 ported '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to 0.94 branch This issue is to port it to trunk -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9420) Math.max() on syncedTillHere lacks synchronization
[
https://issues.apache.org/jira/browse/HBASE-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9420:
--
Status: Open (was: Patch Available)
Math.max() on syncedTillHere lacks synchronization
--
Key: HBASE-9420
URL: https://issues.apache.org/jira/browse/HBASE-9420
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Trivial
Fix For: 0.98.0
Attachments: 9420-v1.txt, 9420-v2.txt
In FSHlog#syncer(), around line 1080:
{code}
this.syncedTillHere = Math.max(this.syncedTillHere, doneUpto);
{code}
Assignment to syncedTillHere after computing max value is not protected by
proper synchronization.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9755) Cannot run classes in hbase-server tests jar from command line
[ https://issues.apache.org/jira/browse/HBASE-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794291#comment-13794291 ] Nick Dimiduk commented on HBASE-9755: - Works for me too. +1 Cannot run classes in hbase-server tests jar from command line --- Key: HBASE-9755 URL: https://issues.apache.org/jira/browse/HBASE-9755 Project: HBase Issue Type: Bug Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.98.0, 0.96.1 Attachments: hbase-9755_v1.patch cached_classpath.txt no longer contains references to hbase-server-version-tests.jar. This prevents to run classes under hbase-server/src/test from bin/hbase script. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794304#comment-13794304 ] Sergey Shelukhin commented on HBASE-5487: - [~jxiang] by janitor, I mean not timeout monitor, but something picking up timeouts of non-master ops like open. It's a rare case and probably never happens in int tests, but there can be a case where RS is taking too long to open. Generic framework for Master-coordinated tasks -- Key: HBASE-5487 URL: https://issues.apache.org/jira/browse/HBASE-5487 Project: HBase Issue Type: New Feature Components: master, regionserver, Zookeeper Affects Versions: 0.94.0 Reporter: Mubarak Seyed Priority: Critical Attachments: Region management in Master.pdf Need a framework to execute master-coordinated tasks in a fault-tolerant manner. Master-coordinated tasks such as online-scheme change and delete-range (deleting region(s) based on start/end key) can make use of this framework. The advantages of framework are 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for master-coordinated tasks 2. Ability to abstract the common functions across Master - ZK and RS - ZK 3. Easy to plugin new master-coordinated tasks without adding code to core components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9692: Component/s: (was: test) TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9692: Issue Type: Test (was: Bug) TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9692: Component/s: test TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Bug Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9692: Fix Version/s: 0.96.1 0.98.0 TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9692: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794307#comment-13794307 ] Nick Dimiduk commented on HBASE-9692: - Committed to 0.96 and trunk. Thanks for the patch JMS. TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794318#comment-13794318 ] Jimmy Xiang commented on HBASE-5487: I see. Generic framework for Master-coordinated tasks -- Key: HBASE-5487 URL: https://issues.apache.org/jira/browse/HBASE-5487 Project: HBase Issue Type: New Feature Components: master, regionserver, Zookeeper Affects Versions: 0.94.0 Reporter: Mubarak Seyed Priority: Critical Attachments: Region management in Master.pdf Need a framework to execute master-coordinated tasks in a fault-tolerant manner. Master-coordinated tasks such as online-scheme change and delete-range (deleting region(s) based on start/end key) can make use of this framework. The advantages of framework are 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for master-coordinated tasks 2. Ability to abstract the common functions across Master - ZK and RS - ZK 3. Easy to plugin new master-coordinated tasks without adding code to core components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794324#comment-13794324
]
Nick Dimiduk commented on HBASE-9570:
-
Forgive my JRuby ignorance but this patch looks like it's unwrapping
TableNotFound, NoSuchColumnFamily, and TableExists exceptions an extra time. Is
that intentional?
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Priority: Minor
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-7379) Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk
[ https://issues.apache.org/jira/browse/HBASE-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7379: - Resolution: Won't Fix Status: Resolved (was: Patch Available) Resolving as 'wont fix' per Ted comment above that it is no longer needed. Port '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to trunk --- Key: HBASE-7379 URL: https://issues.apache.org/jira/browse/HBASE-7379 Project: HBase Issue Type: Sub-task Reporter: Ted Yu Priority: Minor Attachments: 7379-trunk.txt, 7379-trunk-v2.txt HBASE-6728 ported '[89-fb] prevent OOM possibility due to per connection responseQueue being unbounded' to 0.94 branch This issue is to port it to trunk -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794330#comment-13794330
]
Nick Dimiduk commented on HBASE-9570:
-
I take it back, this appears to work as previously.
{noformat}
hbase(main):001:0 create 'foo', 'f'
0 row(s) in 6.5330 seconds
= Hbase::Table - foo
hbase(main):002:0 create 'foo', 'f'
ERROR: Table already exists: foo!
...
{noformat}
Will commit later today unless someone objects.
[~stack] do you want this in 0.96?
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Priority: Minor
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9747) PrefixFilter with OR condition gives wrong results
[
https://issues.apache.org/jira/browse/HBASE-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794334#comment-13794334
]
Aditya Kishore commented on HBASE-9747:
---
Thanks Julian and Deepa for verifying the patch.
PrefixFilter with OR condition gives wrong results
--
Key: HBASE-9747
URL: https://issues.apache.org/jira/browse/HBASE-9747
Project: HBase
Issue Type: Bug
Components: Filters
Affects Versions: 0.95.2, 0.94.9
Reporter: Deepa Remesh
Assignee: Aditya Kishore
Attachments: HBASE-9747_0.94.patch, HBASE-9747.patch
PrefixFilter when used with a SingleColumnValueFilter with an OR condition
gives wrong results. In below example, each filter when evaluated separately
gives 1 row each. The OR condition with the two filters gives 3 rows instead
of 2. Repro below:
create 't1', 'f1'
put 't1','a1','f1:q2','111'
put 't1','b1','f1:q1','112'
put 't1','c1','f1:q1','113'
hbase(main):020:0 scan 't1', {FILTER = PrefixFilter ('b') OR
SingleColumnValueFilter('f1', 'q1', =, 'binary:113')}
ROWCOLUMN+CELL
a1column=f1:q2,
timestamp=1381468905492, value=111
b1column=f1:q1,
timestamp=1381468905518, value=112
c1column=f1:q1,
timestamp=1381468905549, value=113
3 row(s) in 0.1020 seconds
hbase(main):021:0 scan 't1', {FILTER = PrefixFilter ('b')}
ROWCOLUMN+CELL
b1column=f1:q1,
timestamp=1381468905518, value=112
1 row(s) in 0.0150 seconds
hbase(main):002:0 scan 't1', {FILTER = SingleColumnValueFilter('f1', 'q1',
=, 'binary:113')}
ROWCOLUMN+CELL
c1column=f1:q1,
timestamp=1381469178679, value=113
1 row(s) in 0.0140 seconds
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-9570:
Fix Version/s: 0.98.0
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794341#comment-13794341
]
stack commented on HBASE-9570:
--
[~ndimiduk] please. usability.
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[
https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794354#comment-13794354
]
Hadoop QA commented on HBASE-9758:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608301/trunk-9758.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7537//console
This message is automatically generated.
Log missing rows and their regions if ITBLL fails
-
Key: HBASE-9758
URL: https://issues.apache.org/jira/browse/HBASE-9758
Project: HBase
Issue Type: Task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: trunk-9758.patch
Currently, when ITBLL fails, the log only shows how many rows are missing. We
have to go to the MR log to find the rows. The row key is some binary string
that is hard to map to a region. It will be helpful to log the missing rows
in the ITBLL log and the corresponding regions that hold these rows.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-9562:
---
Attachment: HBASE-9562-addendum.patch
I'm late on this one, but we've a method FSUtils.setFsDefault() that set both
(fs.default.name/fs.defaultFS) properties... also to get the root dir we should
use FSUtils.getRootDir() that returns the qualified path needed by the
FileSystem class.
(attached a patch that uses the methods in FSUtils)
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9753) Excessive readpoints checks in MemstoreScanner
[ https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9753: -- Attachment: 9753-v1.txt Something like this ? Excessive readpoints checks in MemstoreScanner -- Key: HBASE-9753 URL: https://issues.apache.org/jira/browse/HBASE-9753 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Vladimir Rodionov Attachments: 9753-v1.txt Brought up by [~vrodionov] on the mailing list. See also HBASE-9751. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-9570:
Assignee: Yang Wang
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Assignee: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794396#comment-13794396
]
stack commented on HBASE-9562:
--
+1 for 0.96 and trunk.
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794395#comment-13794395
]
stack commented on HBASE-9562:
--
This looks like what I tested already. +1
Thanks for fixing what I broke [~mbertozzi]
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794413#comment-13794413 ] Hudson commented on HBASE-9692: --- SUCCESS: Integrated in HBase-TRUNK #4614 (See [https://builds.apache.org/job/HBase-TRUNK/4614/]) HBASE-9692 TestOrderedBytes doesn't display the arrays correctly (ndimiduk: rev 1531992) * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9758: --- Resolution: Fixed Fix Version/s: 0.96.1 0.98.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Integrated into trunk and 0.96. Thanks Stack for the review. Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794493#comment-13794493 ] Hudson commented on HBASE-9692: --- SUCCESS: Integrated in hbase-0.96 #138 (See [https://builds.apache.org/job/hbase-0.96/138/]) HBASE-9692 TestOrderedBytes doesn't display the arrays correctly (ndimiduk: rev 1531997) * /hbase/branches/0.96/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9748: --- Status: Patch Available (was: Open) Address outstanding comments raised for HBASE-9696 -- Key: HBASE-9748 URL: https://issues.apache.org/jira/browse/HBASE-9748 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9748.patch This is a follow-up issue of HBASE-9696. There are some outstanding review comments in https://reviews.apache.org/r/14470/ that haven't been addressed. I will address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9748: --- Attachment: trunk-9748.patch Attached patch addressed outstanding comments raised in reviewing patch for HBASE-9696. The changes are 1, fixed some variable name/method name to avoid confusing such as regionName - encodedName; 2, added/fixed some comments; 3, changed ZKAssign#deleteNode to take additional parameter (target servername) to be certain we are deleting the right znode. Address outstanding comments raised for HBASE-9696 -- Key: HBASE-9748 URL: https://issues.apache.org/jira/browse/HBASE-9748 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9748.patch This is a follow-up issue of HBASE-9696. There are some outstanding review comments in https://reviews.apache.org/r/14470/ that haven't been addressed. I will address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-7544) Transparent table/CF encryption
[
https://issues.apache.org/jira/browse/HBASE-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-7544:
--
Attachment: 7544p4.patch
7544p3.patch
7544p2.patch
7544p1.patch
Rebase against latest trunk.
Patch '7544p4.patch' adds an encrypting protobuf WAL, currently missing support
for dictionary compression but will add that after more testing. Another easy
planned addition here is selective encryption of only the WALEdits for
encrypted families.
Added some unit tests, notably one that confirms if hbck is run on the secure
enclave with access to key material (implicitly using the same configuration as
for regionservers) it can handle encrypted HFiles.
Also note that an ASL licensed open source accelerated JCE codec for AES in CTR
mode is available at https://github.com/intel-hadoop/project-diceros . Will be
used if installed and hbase.crypto.algorithm.aes.provider=DC. This is not
required for HBASE-7544 but will substantially reduce the latency and CPU cost
introduced by encryption compared to the default AES codec that ships with the
Oracle/OpenJDK JRE.
Transparent table/CF encryption
---
Key: HBASE-7544
URL: https://issues.apache.org/jira/browse/HBASE-7544
Project: HBase
Issue Type: New Feature
Components: HFile, io
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.98.0
Attachments: 7544p1.patch, 7544p1.patch, 7544p2.patch, 7544p2.patch,
7544p3.patch, 7544p3.patch, 7544p4.patch, historical-7544.patch,
historical-7544.pdf, historical-shell.patch
Introduce transparent encryption of HBase on disk data.
Depends on a separate contribution of an encryption codec framework to Hadoop
core and an AES-NI (native code) codec. This is work done in the context of
MAPREDUCE-4491 but I'd gather there will be additional JIRAs for common and
HDFS parts of it.
Requirements:
- Transparent encryption at the CF or table level
- Protect against all data leakage from files at rest
- Two-tier key architecture for consistency with best practices for this
feature in the RDBMS world
- Built-in key management
- Flexible and non-intrusive key rotation
- Mechanisms not exposed to or modifiable by users
- Hardware security module integration (via Java KeyStore)
- HBCK support for transparently encrypted files (+ plugin architecture for
HBCK)
Additional goals:
- Shell support for administrative functions
- Avoid performance impact for the null crypto codec case
- Play nicely with other changes underway: in HFile, block coding, etc.
We're aiming for rough parity with Oracle's transparent tablespace encryption
feature, described in
http://www.oracle.com/technetwork/database/owp-security-advanced-security-11gr-133411.pdf
as
{quote}
“Transparent Data Encryption uses a 2-tier key architecture for flexible and
non-intrusive key rotation and least operational and performance impact: Each
application table with at least one encrypted column has its own table key,
which is applied to all encrypted columns in that table. Equally, each
encrypted tablespace has its own tablespace key. Table keys are stored in the
data dictionary of the database, while tablespace keys are stored in the
header of the tablespace and additionally, the header of each underlying OS
file that makes up the tablespace. Each of these keys is encrypted with the
TDE master encryption key, which is stored outside of the database in an
external security module: either the Oracle Wallet (a PKCS#12 formatted file
that is encrypted using a passphrase supplied either by the designated
security administrator or DBA during setup), or a Hardware Security Module
(HSM) device for higher assurance […]”
{quote}
Further design details forthcoming in a design document and patch as soon as
we have all of the clearances in place.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-3727) MultiHFileOutputFormat
[ https://issues.apache.org/jira/browse/HBASE-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794535#comment-13794535 ] Andrew Purtell commented on HBASE-3727: --- The patch on this issue is about two years stale. This should be redone if needed. MultiHFileOutputFormat -- Key: HBASE-3727 URL: https://issues.apache.org/jira/browse/HBASE-3727 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Priority: Minor Attachments: MultiHFileOutputFormat.java, MultiHFileOutputFormat.java Like MultiTableOutputFormat, but outputting HFiles. Key is tablename as an IBW. Creates sub-writers (code cut and pasted from HFileOutputFormat) on demand that produce HFiles in per-table subdirectories of the configured output path. Does not currently support partitioning for existing tables / incremental update. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794544#comment-13794544
]
Ted Yu commented on HBASE-9754:
---
w.r.t. the StoreScanner constructors, there're two which don't have Store
argument. These two constructors are for testing.
The other constructors accept Store argument which has reference to HRegion.
Looks like the approach described above can be simplified.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794546#comment-13794546 ] Hudson commented on HBASE-9758: --- SUCCESS: Integrated in HBase-TRUNK #4615 (See [https://builds.apache.org/job/HBase-TRUNK/4615/]) HBASE-9758 Log missing rows and their regions if ITBLL fails (jxiang: rev 1532048) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script
[ https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794587#comment-13794587 ] Andrew Purtell commented on HBASE-9689: --- +1 patch looks ok, would be good to have coverage in TestShell for this Support using OperationAttributes through shell script -- Key: HBASE-9689 URL: https://issues.apache.org/jira/browse/HBASE-9689 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 Attachments: HBASE-9689_1.patch, HBASE-9689.patch Currently the ruby scripts for Put does not support setting of Operation Attributes through shell. It may be useful in some cases and also for testing. And that would give a full fledged support using shell. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[
https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794607#comment-13794607
]
Hadoop QA commented on HBASE-9748:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608354/trunk-9748.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538//console
This message is automatically generated.
Address outstanding comments raised for HBASE-9696
--
Key: HBASE-9748
URL: https://issues.apache.org/jira/browse/HBASE-9748
Project: HBase
Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9748.patch
This is a follow-up issue of HBASE-9696.
There are some outstanding review comments in
https://reviews.apache.org/r/14470/ that haven't been addressed. I will
address them later in this jira.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9057) Support Tags in ImportTSV tool
[
https://issues.apache.org/jira/browse/HBASE-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794608#comment-13794608
]
Andrew Purtell commented on HBASE-9057:
---
+1
{quote}
+ final static String DEFAULT_TAG_SEPERATOR = \r;
{quote}
I can't think of a better way to do that.
Support Tags in ImportTSV tool
--
Key: HBASE-9057
URL: https://issues.apache.org/jira/browse/HBASE-9057
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-9057.patch
Once we have provision for Tags in the core, the ImportTSV also should
support supplying Tags with the KVs that are to be loaded.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (HBASE-9057) Support Tags in ImportTSV tool
[
https://issues.apache.org/jira/browse/HBASE-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794608#comment-13794608
]
Andrew Purtell edited comment on HBASE-9057 at 10/14/13 11:05 PM:
--
+1 assuming this does not start the unit test flapping. You still seeing that
exception Ram?
{quote}
+ final static String DEFAULT_TAG_SEPERATOR = \r;
{quote}
I can't think of a better way to do that.
was (Author: apurtell):
+1
{quote}
+ final static String DEFAULT_TAG_SEPERATOR = \r;
{quote}
I can't think of a better way to do that.
Support Tags in ImportTSV tool
--
Key: HBASE-9057
URL: https://issues.apache.org/jira/browse/HBASE-9057
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-9057.patch
Once we have provision for Tags in the core, the ImportTSV also should
support supplying Tags with the KVs that are to be loaded.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9748: --- Status: Open (was: Patch Available) Address outstanding comments raised for HBASE-9696 -- Key: HBASE-9748 URL: https://issues.apache.org/jira/browse/HBASE-9748 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9748.patch This is a follow-up issue of HBASE-9696. There are some outstanding review comments in https://reviews.apache.org/r/14470/ that haven't been addressed. I will address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9748) Address outstanding comments raised for HBASE-9696
[ https://issues.apache.org/jira/browse/HBASE-9748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-9748: --- Status: Patch Available (was: Open) This test failure seems unrelated, try hadoop-qa again. Address outstanding comments raised for HBASE-9696 -- Key: HBASE-9748 URL: https://issues.apache.org/jira/browse/HBASE-9748 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9748.patch This is a follow-up issue of HBASE-9696. There are some outstanding review comments in https://reviews.apache.org/r/14470/ that haven't been addressed. I will address them later in this jira. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794544#comment-13794544
]
Ted Yu edited comment on HBASE-9754 at 10/14/13 11:55 PM:
--
w.r.t. the StoreScanner constructors, there're two which don't have Store
argument. These two constructors are for testing.
The other constructors accept Store argument which has reference to HRegion.
was (Author: yuzhih...@gmail.com):
w.r.t. the StoreScanner constructors, there're two which don't have Store
argument. These two constructors are for testing.
The other constructors accept Store argument which has reference to HRegion.
Looks like the approach described above can be simplified.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-9754:
-
Assignee: Ted Yu
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794688#comment-13794688 ] Himanshu Vashishtha commented on HBASE-8741: Thanks for all the reviews, and sorry about the long hiatus on it. I resumed the work on it and attaching the patch for the same. It take care of the above feedbacks. To me, it is an important step in easing out our sequence Id story (disentangle it from region server HLog). In a nut shell, this is the high level idea of this patch: 1) Add a sequence Id attribute to the HRegion class, It is set to openSequenceId when the region is opened. All append operation would use this sequenceId at time of appending to WAL. 2) FSHLog: i) maintain a map of (region : sequenceId) for each WAL. (latestSequenceNums). This keeps track of latest sequence numbers used by a region when appending to the wal. ii) maintain a map of (Path : latestSequenceNums) for each rolled wal file. This is used when determining whether a wal is eligible for archiving or not. A wal is eligible for archiving when all its region has flushed - past the point of their entry in the latestSequenceNums map (as mentioned in i) When rolling a wal, it checks which of the older wals are eligible for archiving. iii) When we run over the maximum number of allowed Wal files, we check the oldest wal file to determine which regions should be flushed so that it could be eligible for archiving. 3) Added test cases in TestHLog to test rolling, archiving and finding memstores to flush to archive a wal. Refactored some test cases to add sequenceId parameter while appending to HLog. 4) Remove the sequenceId support from HLog (sequenceId and helper methods). I tested it on jenkins; ran a patched version on a cluster with insert load while killing region servers and ensuring data is intact. IT tests in progress. Thanks. Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Status: Open (was: Patch Available) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Status: Patch Available (was: Open) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-trunk-v6.patch Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794693#comment-13794693 ] Hudson commented on HBASE-9758: --- SUCCESS: Integrated in hbase-0.96 #139 (See [https://builds.apache.org/job/hbase-0.96/139/]) HBASE-9758 Log missing rows and their regions if ITBLL fails (jxiang: rev 1532051) * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794694#comment-13794694
]
Hudson commented on HBASE-9562:
---
SUCCESS: Integrated in hbase-0.96 #139 (See
[https://builds.apache.org/job/hbase-0.96/139/])
HBASE-9562 Make HLogPE run against configured FS - addendum (mbertozzi: rev
1532087)
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794720#comment-13794720
]
Hudson commented on HBASE-9562:
---
SUCCESS: Integrated in hbase-0.96-hadoop2 #88 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/88/])
HBASE-9562 Make HLogPE run against configured FS - addendum (mbertozzi: rev
1532087)
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794719#comment-13794719 ] Hudson commented on HBASE-9758: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #88 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/88/]) HBASE-9758 Log missing rows and their regions if ITBLL fails (jxiang: rev 1532051) * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794718#comment-13794718 ] Hudson commented on HBASE-9692: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #88 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/88/]) HBASE-9692 TestOrderedBytes doesn't display the arrays correctly (ndimiduk: rev 1531997) * /hbase/branches/0.96/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9758) Log missing rows and their regions if ITBLL fails
[ https://issues.apache.org/jira/browse/HBASE-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794725#comment-13794725 ] Hudson commented on HBASE-9758: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #793 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/793/]) HBASE-9758 Log missing rows and their regions if ITBLL fails (jxiang: rev 1532048) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java Log missing rows and their regions if ITBLL fails - Key: HBASE-9758 URL: https://issues.apache.org/jira/browse/HBASE-9758 Project: HBase Issue Type: Task Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: trunk-9758.patch Currently, when ITBLL fails, the log only shows how many rows are missing. We have to go to the MR log to find the rows. The row key is some binary string that is hard to map to a region. It will be helpful to log the missing rows in the ITBLL log and the corresponding regions that hold these rows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794726#comment-13794726
]
Hudson commented on HBASE-9562:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #793 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/793/])
HBASE-9562 Make HLogPE run against configured FS - addendum (mbertozzi: rev
1532086)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9692) TestOrderedBytes doesn't display the arrays correctly
[ https://issues.apache.org/jira/browse/HBASE-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794724#comment-13794724 ] Hudson commented on HBASE-9692: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #793 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/793/]) HBASE-9692 TestOrderedBytes doesn't display the arrays correctly (ndimiduk: rev 1531992) * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestOrderedBytes.java TestOrderedBytes doesn't display the arrays correctly - Key: HBASE-9692 URL: https://issues.apache.org/jira/browse/HBASE-9692 Project: HBase Issue Type: Test Affects Versions: 0.98.0, 0.96.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9692-v0-trunk.patch, HBASE-9692-v1-trunk.patch, HBASE-9692-v1-trunk.patch TestOrderedBytes should call Arrays.toString(decoded) instead of directly output decoded. Today output will be something like: java.lang.AssertionError: Encoded representations do not preserve natural order: [B@4a8c1dd9, [B@34fbb7cb, ASCENDING: actual array was null We want it to be: java.lang.AssertionError: Encoded representations do not preserve natural order: [], [], ASCENDING: actual array was null -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794729#comment-13794729
]
Hadoop QA commented on HBASE-8741:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608388/HBASE-8741-trunk-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7539//console
This message is automatically generated.
Scope sequenceid to the region rather than regionserver (WAS: Mutations on
Regions in recovery mode might have same sequenceIDs)
Key: HBASE-8741
URL: https://issues.apache.org/jira/browse/HBASE-8741
Project: HBase
Issue Type: Bug
Components: MTTR
Affects Versions: 0.95.1
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Fix For: 0.98.0
Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch,
HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch,
HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch,
HBASE-8741-v5.patch
Currently, when opening a region, we find the maximum sequence ID from all
its HFiles and then set the LogSequenceId of the log (in case the later is at
a small value). This works good in recovered.edits case as we are not writing
to the region until we have replayed all of its previous edits.
With distributed log replay, if we want to enable writes while a region is
under recovery, we need to make sure that the logSequenceId maximum
logSequenceId of the old regionserver. Otherwise, we might have a situation
where new edits have same (or smaller) sequenceIds.
We can store region level information in the WALTrailer, than this scenario
could be avoided by:
a) reading the trailer of the last completed file, i.e., last wal file
which has a trailer and,
b) completely reading the last wal file (this file would not have the
trailer, so it needs to be read completely).
In future, if we switch to multi wal file, we could read the trailer for all
completed WAL files, and reading the
[jira] [Created] (HBASE-9759) IntegrationTestBulkLoad random number collision
Enis Soztutar created HBASE-9759:
Summary: IntegrationTestBulkLoad random number collision
Key: HBASE-9759
URL: https://issues.apache.org/jira/browse/HBASE-9759
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
ITBL failed recently in our test harness. Inspecting the failure made me
believe that the only reason that particular failure might have happened is
that there is a collision in random longs generated by the test.
The test creates 50 mappers by default, and each mapper writes a 500K random
rows starting with row = 0. By default there are 5 iterations.
The check job outputs these counters:
{code}
2013-10-13 07:48:01,134 Map input records=124999751
2013-10-13 07:48:01,134 Map output records=12499
{code}
The number of input records seems fine because
{code}
124999751 = 1 + 5 * (0.5M - 1) * 50
{code}
5 = num iterations, 0.5M = num rows, 50 = num mappers, and 1 is for row =0
which every chain writes to.
Output records should be 125M, however we see one cell missing. Since the map
input records matches expected number of distinct rows, I suspect that row = 0
had a collision.
In one of the generate jobs, we can see that the reducer output count does not
match the reducer input count. Given that we are using KVSortReducer, this
confirms that there is a collision in KeyValues received by this task.
{code}
2013-10-13 06:48:12,738 Reduce input records=7500
2013-10-13 06:48:12,738 Reduce output records=7497
{code}
The count is off by 3 because we are writing 3 columns per row.
My only theory for explaining this is that we had a collision in chainId's or
one of the chains reused row = 0 as the next row.
This is similar to HBASE-8700, however, in this the probability is much much
much lower.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9562) Make HLogPE run against configured FS
[
https://issues.apache.org/jira/browse/HBASE-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794745#comment-13794745
]
Hudson commented on HBASE-9562:
---
FAILURE: Integrated in HBase-TRUNK #4616 (See
[https://builds.apache.org/job/HBase-TRUNK/4616/])
HBASE-9562 Make HLogPE run against configured FS - addendum (mbertozzi: rev
1532086)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
Make HLogPE run against configured FS
-
Key: HBASE-9562
URL: https://issues.apache.org/jira/browse/HBASE-9562
Project: HBase
Issue Type: Sub-task
Components: Performance, wal
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 9562.txt, HBASE-9562-addendum.patch
HLogPE will not pick up the configured fs from *.xml files; always goes
against localfs
Simple patch:
{code}
diff --git
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvalu
index 9c07046..f73258b 100644
---
a/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
+++
b/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
@@ -47,6 +47,7 @@ import org.apache.hadoop.hbase.regionserver.wal.HLog.Entry;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.hadoop.hbase.HConstants;
/**
* This class runs performance benchmarks for {@link HLog}.
@@ -170,8 +171,12 @@ public final class HLogPerformanceEvaluation extends
Configured implements Tool
}
// Run HLog Performance Evaluation
+// First set the fs from configs. Do it for both configs in case we
+// are on hadoop1
+getConf().set(fs.default.name, getConf().get(HConstants.HBASE_DIR));
+getConf().set(fs.defaultFS, getConf().get(HConstants.HBASE_DIR));
FileSystem fs = FileSystem.get(getConf());
-LOG.info( + fs);
+LOG.info(FileSystem: + fs);
try {
if (rootRegionDir == null) {
rootRegionDir =
TEST_UTIL.getDataTestDir(HLogPerformanceEvaluation);
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9754:
--
Attachment: 9754-rp-0.txt
Initial patch.
I ran the following unit tests locally and they passed:
{code}
1164 mt2 -Dtest=TestMemStore
1167 mt2
-Dtest=TestCoprocessorScanPolicy,TestMemStoreChunkPool,TestFSErrorsExposed,TestRegionObserverScannerOpenHook
1173 mt2 -Dtest=TestHRegion
1175 mt2 -Dtest=TestJoinedScanners
{code}
API changes are in classes marked @InterfaceAudience.Private
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794755#comment-13794755
]
Ted Yu commented on HBASE-9754:
---
Patch contains patch for HBASE-9753 as well.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9754:
--
Fix Version/s: 0.98.0
Status: Patch Available (was: Open)
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794757#comment-13794757
]
Ted Yu commented on HBASE-9754:
---
Here is review request: https://reviews.apache.org/r/14642/
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk updated HBASE-9570:
Attachment: HBASE-9570-0.96.01.patch
Patch applied to trunk, not so cleanly to 0.96. I'm not sure why. [~wangy] mind
having a look at this patch for trunk, make sure i didn't screw up something?
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Assignee: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch,
HBASE-9570-0.96.01.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794790#comment-13794790
]
Nick Dimiduk commented on HBASE-9570:
-
Err, make that patch for 0.96.
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Assignee: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch,
HBASE-9570-0.96.01.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794792#comment-13794792 ] Sergey Shelukhin commented on HBASE-5487: - Ok, it's harder than I thought, I don't think I will be done today... but I think I have a clear picture now that covers the above feedback, so I am trying to cover all the failover scenarios and state conflicts. Generic framework for Master-coordinated tasks -- Key: HBASE-5487 URL: https://issues.apache.org/jira/browse/HBASE-5487 Project: HBase Issue Type: New Feature Components: master, regionserver, Zookeeper Affects Versions: 0.94.0 Reporter: Mubarak Seyed Priority: Critical Attachments: Region management in Master.pdf Need a framework to execute master-coordinated tasks in a fault-tolerant manner. Master-coordinated tasks such as online-scheme change and delete-range (deleting region(s) based on start/end key) can make use of this framework. The advantages of framework are 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for master-coordinated tasks 2. Ability to abstract the common functions across Master - ZK and RS - ZK 3. Easy to plugin new master-coordinated tasks without adding code to core components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794818#comment-13794818
]
Hadoop QA commented on HBASE-9754:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608406/9754-rp-0.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7540//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7540//console
This message is automatically generated.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794820#comment-13794820
]
Hadoop QA commented on HBASE-9570:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608415/HBASE-9570-0.96.01.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7541//console
This message is automatically generated.
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Assignee: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch,
HBASE-9570-0.96.01.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9759) IntegrationTestBulkLoad random number collision
[
https://issues.apache.org/jira/browse/HBASE-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-9759:
-
Attachment: hbase-9759_v1.patch
Here is a patch for eliminating chain collision probability. We can apply this
patch to see whether the theory is correct and continue to monitor the test.
Tested the patch on a cluster.
IntegrationTestBulkLoad random number collision
---
Key: HBASE-9759
URL: https://issues.apache.org/jira/browse/HBASE-9759
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9759_v1.patch
ITBL failed recently in our test harness. Inspecting the failure made me
believe that the only reason that particular failure might have happened is
that there is a collision in random longs generated by the test.
The test creates 50 mappers by default, and each mapper writes a 500K random
rows starting with row = 0. By default there are 5 iterations.
The check job outputs these counters:
{code}
2013-10-13 07:48:01,134 Map input records=124999751
2013-10-13 07:48:01,134 Map output records=12499
{code}
The number of input records seems fine because
{code}
124999751 = 1 + 5 * (0.5M - 1) * 50
{code}
5 = num iterations, 0.5M = num rows, 50 = num mappers, and 1 is for row =0
which every chain writes to.
Output records should be 125M, however we see one cell missing. Since the map
input records matches expected number of distinct rows, I suspect that row =
0 had a collision.
In one of the generate jobs, we can see that the reducer output count does
not match the reducer input count. Given that we are using KVSortReducer,
this confirms that there is a collision in KeyValues received by this task.
{code}
2013-10-13 06:48:12,738 Reduce input records=7500
2013-10-13 06:48:12,738 Reduce output records=7497
{code}
The count is off by 3 because we are writing 3 columns per row.
My only theory for explaining this is that we had a collision in chainId's or
one of the chains reused row = 0 as the next row.
This is similar to HBASE-8700, however, in this the probability is much much
much lower.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9759) IntegrationTestBulkLoad random number collision
[
https://issues.apache.org/jira/browse/HBASE-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-9759:
-
Status: Patch Available (was: Open)
IntegrationTestBulkLoad random number collision
---
Key: HBASE-9759
URL: https://issues.apache.org/jira/browse/HBASE-9759
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9759_v1.patch
ITBL failed recently in our test harness. Inspecting the failure made me
believe that the only reason that particular failure might have happened is
that there is a collision in random longs generated by the test.
The test creates 50 mappers by default, and each mapper writes a 500K random
rows starting with row = 0. By default there are 5 iterations.
The check job outputs these counters:
{code}
2013-10-13 07:48:01,134 Map input records=124999751
2013-10-13 07:48:01,134 Map output records=12499
{code}
The number of input records seems fine because
{code}
124999751 = 1 + 5 * (0.5M - 1) * 50
{code}
5 = num iterations, 0.5M = num rows, 50 = num mappers, and 1 is for row =0
which every chain writes to.
Output records should be 125M, however we see one cell missing. Since the map
input records matches expected number of distinct rows, I suspect that row =
0 had a collision.
In one of the generate jobs, we can see that the reducer output count does
not match the reducer input count. Given that we are using KVSortReducer,
this confirms that there is a collision in KeyValues received by this task.
{code}
2013-10-13 06:48:12,738 Reduce input records=7500
2013-10-13 06:48:12,738 Reduce output records=7497
{code}
The count is off by 3 because we are writing 3 columns per row.
My only theory for explaining this is that we had a collision in chainId's or
one of the chains reused row = 0 as the next row.
This is similar to HBASE-8700, however, in this the probability is much much
much lower.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794827#comment-13794827
]
Ted Yu commented on HBASE-9754:
---
I saw the same number of findbugs warnings in:
https://builds.apache.org/job/PreCommit-HBASE-Build/7538/console
which was for HBASE-9748
I think the cause was the following:
{code}
[INFO] Compiling 44 source files to
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-thrift/target/classes
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 90184 bytes for
Arena::Amalloc
# An error report file with more information is saved as:
#
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hs_err_pid8382.log
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/dev-support/test-patch.sh:
line 561: 8382 Aborted $MVN clean package findbugs:findbugs
-D${PROJECT_NAME}PatchProcess -DskipTests /dev/null
{code}
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9570) With AccessDeniedException, HBase shell would be better to just display the error message to be user friendly
[
https://issues.apache.org/jira/browse/HBASE-9570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794849#comment-13794849
]
Hudson commented on HBASE-9570:
---
SUCCESS: Integrated in HBase-TRUNK #4617 (See
[https://builds.apache.org/job/HBase-TRUNK/4617/])
HBASE-9570 With AccessDeniedException, HBase shell would be better to just
display the error message to be user friendly (Yang Wang) (ndimiduk: rev
1532163)
* /hbase/trunk/hbase-shell/src/main/ruby/shell/commands.rb
With AccessDeniedException, HBase shell would be better to just display the
error message to be user friendly
-
Key: HBASE-9570
URL: https://issues.apache.org/jira/browse/HBASE-9570
Project: HBase
Issue Type: Improvement
Components: shell
Reporter: Yang Wang
Assignee: Yang Wang
Priority: Minor
Fix For: 0.98.0
Attachments: HBASE-9570, HBASE-9570.01.patch, HBASE-9570.02.patch,
HBASE-9570-0.96.01.patch
When access unauthorized resource like table, AccessDeniedException will be
thrown. In HBase shell, the error message with stack trace will be displayed
as follows. It would be better to just display the error message avoiding the
stack trace to be user friendly.
{noformat}
ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
permissions for user 'u1' for scanner open on table t1
at
org.apache.hadoop.hbase.security.access.AccessController.preScannerOpen(AccessController.java:1116)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preScannerOpen(RegionCoprocessorHost.java:1293)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3026)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26971)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2083)
at
org.apache.hadoop.hbase.ipc.RpcServer$CallRunner.run(RpcServer.java:1820)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:165)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:41)
at
org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:113)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9753) Excessive readpoints checks in MemstoreScanner
[ https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794850#comment-13794850 ] Lars Hofhansl commented on HBASE-9753: -- That's it? Nice. +1, Let's definitely do this (0.94+). Excessive readpoints checks in MemstoreScanner -- Key: HBASE-9753 URL: https://issues.apache.org/jira/browse/HBASE-9753 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Vladimir Rodionov Attachments: 9753-v1.txt Brought up by [~vrodionov] on the mailing list. See also HBASE-9751. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9137) Add Tag dictionary in WAL compression
[ https://issues.apache.org/jira/browse/HBASE-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-9137: -- Resolution: Fixed Release Note: Tag compression will work when WAL compression is enabled. Set hbase.regionserver.wal.tags.enablecompression false to disable tag compression in WAL. (This property defaults to true) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to Trunk. Thanks for the reviews Add Tag dictionary in WAL compression - Key: HBASE-9137 URL: https://issues.apache.org/jira/browse/HBASE-9137 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: Anoop Sam John Fix For: 0.98.0 Attachments: HBASE-9137.patch, HBASE-9137_V2.patch, HBASE-9137_V3.patch We can add tag dictionary like we have one for rowdictionary, familydictionary. But this has to be done after stabilizing HBASE-7391. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794855#comment-13794855
]
Lars Hofhansl commented on HBASE-9754:
--
Did a quick read through. Looks good. Need to look in more detail.
Using 0 as readPt when unknown is interesting.
Could we use the readPt to replace the hasMVCC stuff I just added in
HBASE-9751? I.e. pass MAX_LONG as readPt when we would have passed
hasMVCC=true, in which case we know we do not need to check the memstoreTs...?
I'd be in favor of committing HBASE-9753 first as that is simple, and
scrutinize this one a bit more.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9759) IntegrationTestBulkLoad random number collision
[
https://issues.apache.org/jira/browse/HBASE-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794859#comment-13794859
]
stack commented on HBASE-9759:
--
+1 on trying the patch. How does it provent collision (I did not review
closely).
If you do a select on row 0, does it have more versions than other rows.
What is to prevent our clashing randomly on another row? Because our randoms
generation is within a fixed range per iteration?
IntegrationTestBulkLoad random number collision
---
Key: HBASE-9759
URL: https://issues.apache.org/jira/browse/HBASE-9759
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9759_v1.patch
ITBL failed recently in our test harness. Inspecting the failure made me
believe that the only reason that particular failure might have happened is
that there is a collision in random longs generated by the test.
The test creates 50 mappers by default, and each mapper writes a 500K random
rows starting with row = 0. By default there are 5 iterations.
The check job outputs these counters:
{code}
2013-10-13 07:48:01,134 Map input records=124999751
2013-10-13 07:48:01,134 Map output records=12499
{code}
The number of input records seems fine because
{code}
124999751 = 1 + 5 * (0.5M - 1) * 50
{code}
5 = num iterations, 0.5M = num rows, 50 = num mappers, and 1 is for row =0
which every chain writes to.
Output records should be 125M, however we see one cell missing. Since the map
input records matches expected number of distinct rows, I suspect that row =
0 had a collision.
In one of the generate jobs, we can see that the reducer output count does
not match the reducer input count. Given that we are using KVSortReducer,
this confirms that there is a collision in KeyValues received by this task.
{code}
2013-10-13 06:48:12,738 Reduce input records=7500
2013-10-13 06:48:12,738 Reduce output records=7497
{code}
The count is off by 3 because we are writing 3 columns per row.
My only theory for explaining this is that we had a collision in chainId's or
one of the chains reused row = 0 as the next row.
This is similar to HBASE-8700, however, in this the probability is much much
much lower.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9759) IntegrationTestBulkLoad random number collision
[
https://issues.apache.org/jira/browse/HBASE-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794860#comment-13794860
]
stack commented on HBASE-9759:
--
Why this
-s.setBatch(100);
+s.setBatch(1000);
IntegrationTestBulkLoad random number collision
---
Key: HBASE-9759
URL: https://issues.apache.org/jira/browse/HBASE-9759
Project: HBase
Issue Type: Bug
Components: test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.98.0, 0.96.1
Attachments: hbase-9759_v1.patch
ITBL failed recently in our test harness. Inspecting the failure made me
believe that the only reason that particular failure might have happened is
that there is a collision in random longs generated by the test.
The test creates 50 mappers by default, and each mapper writes a 500K random
rows starting with row = 0. By default there are 5 iterations.
The check job outputs these counters:
{code}
2013-10-13 07:48:01,134 Map input records=124999751
2013-10-13 07:48:01,134 Map output records=12499
{code}
The number of input records seems fine because
{code}
124999751 = 1 + 5 * (0.5M - 1) * 50
{code}
5 = num iterations, 0.5M = num rows, 50 = num mappers, and 1 is for row =0
which every chain writes to.
Output records should be 125M, however we see one cell missing. Since the map
input records matches expected number of distinct rows, I suspect that row =
0 had a collision.
In one of the generate jobs, we can see that the reducer output count does
not match the reducer input count. Given that we are using KVSortReducer,
this confirms that there is a collision in KeyValues received by this task.
{code}
2013-10-13 06:48:12,738 Reduce input records=7500
2013-10-13 06:48:12,738 Reduce output records=7497
{code}
The count is off by 3 because we are writing 3 columns per row.
My only theory for explaining this is that we had a collision in chainId's or
one of the chains reused row = 0 as the next row.
This is similar to HBASE-8700, however, in this the probability is much much
much lower.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794863#comment-13794863
]
Ted Yu commented on HBASE-9754:
---
bq. scrutinize this one a bit more.
Take your time.
Depending on how long the review takes, we can decide which one to commit
first. HBASE-9753 is a subset of this.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9753) Excessive readpoints checks in MemstoreScanner
[ https://issues.apache.org/jira/browse/HBASE-9753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-9753: -- Status: Patch Available (was: Open) Excessive readpoints checks in MemstoreScanner -- Key: HBASE-9753 URL: https://issues.apache.org/jira/browse/HBASE-9753 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Vladimir Rodionov Attachments: 9753-v1.txt Brought up by [~vrodionov] on the mailing list. See also HBASE-9751. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5487) Generic framework for Master-coordinated tasks
[ https://issues.apache.org/jira/browse/HBASE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794862#comment-13794862 ] Eric Newton commented on HBASE-5487: Accumulo does manage tablet (region) assignment tracking through the metadata table, and further, uses a distributed state machine to scale up a little beyond a single master node. I have been meaning to write it up, but I've not had a chance. I've not kept up with every HBase improvement, so I don't know if it is pertinent... the accumulo metadata table is typically spread out over 50 - 100% of the available tablet (region) servers. Still, the metadata table, and especially the root table(t), is subject to hot-spotting on large map/reduce jobs where hundreds (or thousands) of clients are learning tablet locations at the same time. Block caching is important, but at some point massive numbers of simultaneous RPC requests to a single node cause delays, or even timeouts and failures. But using accumulo to store accumulo state has scaled well. Accumulo has 2 frameworks for master tasks: * master general state processing: a table should be online, assignments are recorded and servers repeatedly informed * FATE processing, where multi-stage operations are saved, executed and progress is re-recorded The first is general maintenance: keeping the system running. Tablets are assigned, unassigned and in-general balanced. The second allows for temporal deviance: tablets are taken offline for a merge, for example. The step-by-step allocation of resources and state are walked, each step recording progress in zookeeper. Generic framework for Master-coordinated tasks -- Key: HBASE-5487 URL: https://issues.apache.org/jira/browse/HBASE-5487 Project: HBase Issue Type: New Feature Components: master, regionserver, Zookeeper Affects Versions: 0.94.0 Reporter: Mubarak Seyed Priority: Critical Attachments: Region management in Master.pdf Need a framework to execute master-coordinated tasks in a fault-tolerant manner. Master-coordinated tasks such as online-scheme change and delete-range (deleting region(s) based on start/end key) can make use of this framework. The advantages of framework are 1. Eliminate repeated code in Master, ZooKeeper tracker and Region-server for master-coordinated tasks 2. Ability to abstract the common functions across Master - ZK and RS - ZK 3. Easy to plugin new master-coordinated tasks without adding code to core components -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794866#comment-13794866 ] Himanshu Vashishtha commented on HBASE-8741: I remember this, heapsize failing on apache pre-commit m/c. Let me take a look at it. Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Consider eliminating threadlocals from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794865#comment-13794865
]
Ted Yu commented on HBASE-9754:
---
bq. Could we use the readPt to replace the hasMVCC stuff
Looking at the code:
{code}
+ public boolean hasMVCCInfo() {
+return includesMemstoreTS decodeMemstoreTS;
{code}
I think they represent different things.
Consider eliminating threadlocals from MVCC code
Key: HBASE-9754
URL: https://issues.apache.org/jira/browse/HBASE-9754
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Ted Yu
Fix For: 0.98.0
Attachments: 9754-rp-0.txt
Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
Currently we use ThreadLocals to communicate the current readpoint between a
RegionScanner and the Store\{File}Scanner's down the stack.
Since ThreadLocals are not cheap we should consider whether it is possible to
pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9689) Support using OperationAttributes through shell script
[
https://issues.apache.org/jira/browse/HBASE-9689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13794867#comment-13794867
]
Anoop Sam John commented on HBASE-9689:
---
Looks good
{code}
+ # Apply user metadata to table/column descriptor
+def set_scan_attributes(scan, attributes)
{code}
Correct the comment pls. Seems copy paste :)
set_scan_attributes, set_get_attributes, set_put_attributes seems same code
repeating. We can make this one def?
Support using OperationAttributes through shell script
--
Key: HBASE-9689
URL: https://issues.apache.org/jira/browse/HBASE-9689
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.0
Attachments: HBASE-9689_1.patch, HBASE-9689.patch
Currently the ruby scripts for Put does not support setting of Operation
Attributes through shell.
It may be useful in some cases and also for testing. And that would give a
full fledged support using shell.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
