[jira] [Commented] (HBASE-15952) Bulk load data replication is not working when RS user does not have permission on hfile-refs node
[ https://issues.apache.org/jira/browse/HBASE-15952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320049#comment-15320049 ] Ashish Singhi commented on HBASE-15952: --- bq. Is the above test failure reproducible ? No. I ran this tests 3 times on branch-1 and not able to reproducible. Also I don't think the test failure is related anywhere to my patch. > Bulk load data replication is not working when RS user does not have > permission on hfile-refs node > -- > > Key: HBASE-15952 > URL: https://issues.apache.org/jira/browse/HBASE-15952 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.0 >Reporter: Ashish Singhi >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-15952.branch-1.v1.patch, HBASE-15952.patch, > HBASE-15952.v1.patch > > > In our recent testing in secure cluster we found that when a RS user does not > have permission on hfile-refs znode then RS was failing to replicate the bulk > loaded data as the hfile-refs znode is created by hbase client and RS user > may not have permission to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[ https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320048#comment-15320048 ] huaxiang sun commented on HBASE-15959: -- The old store files are not archived right after compaction, compaction.discharger does that periodically. After the first major compaction, there are 6 store files, 5 smaller ones, one with everything. Since compaction.discharger does not kick in yet, at that time, region.initialize() will open these files again, I think at that time, it messed up, some cells are from the old hfile which has mob reference, some from the final one, which does not have the reference cell. I will upload a patch shortly, thanks. > Fix flaky test TestRegionServerMetrics.testMobMetrics > - > > Key: HBASE-15959 > URL: https://issues.apache.org/jira/browse/HBASE-15959 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: huaxiang sun > > It flakes > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460]. > There are two weird things i identified: > 1. In second compaction, > [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173] > has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from > earlier compaction. Probably because mob cleaner doesn't clean old hfiles. > Does this needs fixing? > 2. Across runs, same cell (ie. same key) may or may not be considered mob > reference cell. > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213]. > This at least happens with row keys 0 - 4 (which got compacted earlier). > [~jmhsieh] Any ideas why this would happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Attachment: HBASE-14743.003.patch > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-14743.001.patch, HBASE-14743.002.patch, > HBASE-14743.003.patch > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks
[
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320035#comment-15320035
]
Hudson commented on HBASE-15600:
FAILURE: Integrated in HBase-1.4 #202 (See
[https://builds.apache.org/job/HBase-1.4/202/])
HBASE-15600 Add provision for adding mutations to memstore or able to (enis:
rev 602b2839c688b96a57389bbfe20cd0e2073f22e6)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverForAddingMutationsFromCoprocessors.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MiniBatchOperationInProgress.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiRowMutationProcessor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> Add provision for adding mutations to memstore or able to write to same
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Labels: phoenix
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch,
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch,
> hbase-15600_v5.patch, hbase-15600_v6.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region
> from coprocessors but writing from batchMutate API is not allowed because of
> mvcc.
> Raised PHOENIX-2742 to discuss any alternative way to write to the same
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set
> wal edits in MiniBatchOperationInProgress.
> {noformat}
> /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
> public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
> }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as
> well. Or else we should provide the batch mutation API allow write in batch
> mutate coprocessors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14743) Add metrics around HeapMemoryManager
[
https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320011#comment-15320011
]
Hadoop QA commented on HBASE-14743:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 54s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestMetricsHeapMemoryManager |
| | hadoop.hbase.regionserver.TestHeapMemoryManager |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
[jira] [Commented] (HBASE-15950) Fix memstore size estimates to be more tighter
[
https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320002#comment-15320002
]
Hadoop QA commented on HBASE-15950:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 16s {color}
| {color:red} hbase-common-jdk1.8.0 with JDK v1.8.0 generated 3 new + 26
unchanged - 0 fixed = 29 total (was 26) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 4s {color}
| {color:red} hbase-common-jdk1.7.0_79 with JDK v1.7.0_79 generated 3 new + 26
unchanged - 0 fixed = 29 total (was 26) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 51s
{color} | {color:red} hbase-common-jdk1.7.0_79 with JDK v1.7.0_79 generated 1
new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 36s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green}
[jira] [Commented] (HBASE-14743) Add metrics around HeapMemoryManager
[
https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319996#comment-15319996
]
Hadoop QA commented on HBASE-14743:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 27s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 37s
{color} | {color:red} hbase-hadoop-compat-jdk1.8.0 with JDK v1.8.0 generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m 34s
{color} | {color:red} hbase-hadoop-compat-jdk1.7.0_79 with JDK v1.7.0_79
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 17s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} |
[jira] [Commented] (HBASE-15981) Stripe and Date-tiered compactions inaccurately suggest disabling table in docs
[
https://issues.apache.org/jira/browse/HBASE-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319981#comment-15319981
]
Nick Dimiduk commented on HBASE-15981:
--
Sure. Filed HBASE-15989.
> Stripe and Date-tiered compactions inaccurately suggest disabling table in
> docs
> ---
>
> Key: HBASE-15981
> URL: https://issues.apache.org/jira/browse/HBASE-15981
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15981.patch
>
>
> {{hbase.online.schema.update.enable}} is set to true in later versions of
> HBase. However, the documentation in the book around configuration Stripe and
> Date-tiered compactions instruct the user to disable a table before changing
> configuration. According to Enis on the user list, this is not necessary:
> {quote}
> Disabling the table should not be needed. From the stripe compaction
> perspective, deploying this in a disabled table versus in an online alter
> table is not different at all. The "hbase.online.schema.update.enable"
> property was fixing some possible race conditions that were fixed long time
> ago.
> {quote}
> We should update the book to remove this inaccurate messaging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15989) Remove hbase.online.schema.update.enable
[ https://issues.apache.org/jira/browse/HBASE-15989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319980#comment-15319980 ] Nick Dimiduk commented on HBASE-15989: -- FYI [~enis] [~bbeaudreault] > Remove hbase.online.schema.update.enable > > > Key: HBASE-15989 > URL: https://issues.apache.org/jira/browse/HBASE-15989 > Project: HBase > Issue Type: Task >Reporter: Nick Dimiduk > > Pre discussion on HBASE-15981, the configuration > hbase.online.schema.update.enable is an artifact of a bygone era. Lets rip it > out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15989) Remove hbase.online.schema.update.enable
Nick Dimiduk created HBASE-15989: Summary: Remove hbase.online.schema.update.enable Key: HBASE-15989 URL: https://issues.apache.org/jira/browse/HBASE-15989 Project: HBase Issue Type: Task Reporter: Nick Dimiduk Pre discussion on HBASE-15981, the configuration hbase.online.schema.update.enable is an artifact of a bygone era. Lets rip it out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[ https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319960#comment-15319960 ] Jingcheng Du commented on HBASE-15959: -- bq. (Just to be sure, reference cells = mob cells, right?) In my context, reference cells are cells in store files, they are linked to cells in mob files, and the cells in mob files are mob files:). Sorry for not clarifying things clearly. > Fix flaky test TestRegionServerMetrics.testMobMetrics > - > > Key: HBASE-15959 > URL: https://issues.apache.org/jira/browse/HBASE-15959 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: huaxiang sun > > It flakes > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460]. > There are two weird things i identified: > 1. In second compaction, > [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173] > has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from > earlier compaction. Probably because mob cleaner doesn't clean old hfiles. > Does this needs fixing? > 2. Across runs, same cell (ie. same key) may or may not be considered mob > reference cell. > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213]. > This at least happens with row keys 0 - 4 (which got compacted earlier). > [~jmhsieh] Any ideas why this would happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Attachment: HBASE-14743.002.patch > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-14743.001.patch, HBASE-14743.002.patch > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Attachment: (was: HBASE-14743.002.patch) > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-14743.001.patch > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[ https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319953#comment-15319953 ] Jingcheng Du commented on HBASE-15959: -- Maybe the automatic run of the region compaction affects the final results, reducing the number of store files might fix this. But as Appy said, the number of store files is still 5 after the 1st compaction, the old store files are not archived, I tested this, this is not only for mob, it is also for non-mob tables. I'll file another JIRA to fix this. Thanks. > Fix flaky test TestRegionServerMetrics.testMobMetrics > - > > Key: HBASE-15959 > URL: https://issues.apache.org/jira/browse/HBASE-15959 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: huaxiang sun > > It flakes > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460]. > There are two weird things i identified: > 1. In second compaction, > [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173] > has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from > earlier compaction. Probably because mob cleaner doesn't clean old hfiles. > Does this needs fixing? > 2. Across runs, same cell (ie. same key) may or may not be considered mob > reference cell. > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213]. > This at least happens with row keys 0 - 4 (which got compacted earlier). > [~jmhsieh] Any ideas why this would happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks
[
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319951#comment-15319951
]
Anoop Sam John commented on HBASE-15600:
As per discussion in another Jira, it was concluded that these kind of new
features should not be committed to patch branches. So am not sure. Ping RMs
> Add provision for adding mutations to memstore or able to write to same
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Labels: phoenix
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch,
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch,
> hbase-15600_v5.patch, hbase-15600_v6.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region
> from coprocessors but writing from batchMutate API is not allowed because of
> mvcc.
> Raised PHOENIX-2742 to discuss any alternative way to write to the same
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set
> wal edits in MiniBatchOperationInProgress.
> {noformat}
> /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
> public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
> }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as
> well. Or else we should provide the batch mutation API allow write in batch
> mutate coprocessors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks
[
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319949#comment-15319949
]
Rajeshbabu Chintaguntla commented on HBASE-15600:
-
[~enis] I need this in 0.98 , 1.0 , 1.1 and 1.2 branches for phoenix.
> Add provision for adding mutations to memstore or able to write to same
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Labels: phoenix
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch,
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch,
> hbase-15600_v5.patch, hbase-15600_v6.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region
> from coprocessors but writing from batchMutate API is not allowed because of
> mvcc.
> Raised PHOENIX-2742 to discuss any alternative way to write to the same
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set
> wal edits in MiniBatchOperationInProgress.
> {noformat}
> /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
> public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
> }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as
> well. Or else we should provide the batch mutation API allow write in batch
> mutate coprocessors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14743) Add metrics around HeapMemoryManager
[ https://issues.apache.org/jira/browse/HBASE-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan updated HBASE-14743: -- Attachment: HBASE-14743.002.patch > Add metrics around HeapMemoryManager > > > Key: HBASE-14743 > URL: https://issues.apache.org/jira/browse/HBASE-14743 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Reid Chan >Priority: Minor > Attachments: HBASE-14743.001.patch, HBASE-14743.002.patch > > > it would be good to know how many invocations there have been. > How many decided to expand memstore. > How many decided to expand block cache. > How many decided to do nothing. > etc. > When that's done use those metrics to clean up the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks
[
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319939#comment-15319939
]
Enis Soztutar commented on HBASE-15600:
---
[~chrajeshbab...@gmail.com] for Phoenix to start using this, do we need this in
1.1 and 1.2 as well? Normally this is a new feature, but maybe we can make a
case.
> Add provision for adding mutations to memstore or able to write to same
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Labels: phoenix
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch,
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch,
> hbase-15600_v5.patch, hbase-15600_v6.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region
> from coprocessors but writing from batchMutate API is not allowed because of
> mvcc.
> Raised PHOENIX-2742 to discuss any alternative way to write to the same
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set
> wal edits in MiniBatchOperationInProgress.
> {noformat}
> /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
> public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
> }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as
> well. Or else we should provide the batch mutation API allow write in batch
> mutate coprocessors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks
[
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319936#comment-15319936
]
Enis Soztutar commented on HBASE-15600:
---
bq. Can change on commit if you agree Enis. +1
Thanks Anoop. I've pushed this to 1.3+ with this change. Keeping open for 0.98
patch.
> Add provision for adding mutations to memstore or able to write to same
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Labels: phoenix
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch,
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch,
> hbase-15600_v5.patch, hbase-15600_v6.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region
> from coprocessors but writing from batchMutate API is not allowed because of
> mvcc.
> Raised PHOENIX-2742 to discuss any alternative way to write to the same
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set
> wal edits in MiniBatchOperationInProgress.
> {noformat}
> /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
> public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
> }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as
> well. Or else we should provide the batch mutation API allow write in batch
> mutate coprocessors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15714) We are calling checkRow() twice in doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-15714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-15714:
--
Fix Version/s: 1.3.0
> We are calling checkRow() twice in doMiniBatchMutation()
>
>
> Key: HBASE-15714
> URL: https://issues.apache.org/jira/browse/HBASE-15714
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15714-branch-1.patch, HBASE-15714.patch,
> HBASE-15714_v1.patch, HBASE-15714_v2.patch
>
>
> In {{multi()}} -> {{doMiniBatchMutation()}} code path, we end up calling
> {{checkRow()}} twice, once from {{checkBatchOp()}} and once from
> {{getRowLock()}}.
> See [~anoop.hbase]'s comments at
> https://issues.apache.org/jira/browse/HBASE-15600?focusedCommentId=15257636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15257636.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[ https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319926#comment-15319926 ] Appy commented on HBASE-15959: -- bq. region.compact doesn't compact mob files, I think this is not related with cleaners. If it's moving cells from mob to store, doesn't that mean it's compacting mob files too. [Here|https://github.com/apache/hbase/blob/7efb9edecbdf8b35046230575d504e4caeb80f34/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213], we are moving mob cells to store or new writer. You're right, first two parts are very clear what they are doing. Set the thresholds, expect things to move around. bq. In the 2nd compaction, the threshold is changed to 0, and all cells are moved to mob files from store files. And in the second scan, the cells are the reference ones. Cells 0-4 got moved to store file in the first compaction. And that's why in the [second scan|https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L441], there is [no mob cell|https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L445]. And we write cells 5-9 before [threshold is ste to 0|https://github.com/apache/hbase/blob/master/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L453], so they should be non-mob cells too. So when the second compaction kicks, I would expect them to be non-reference cells. It's only after the compaction has finished, that they'll become the reference cells again. (Just to be sure, reference cells = mob cells, right?) > Fix flaky test TestRegionServerMetrics.testMobMetrics > - > > Key: HBASE-15959 > URL: https://issues.apache.org/jira/browse/HBASE-15959 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: huaxiang sun > > It flakes > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460]. > There are two weird things i identified: > 1. In second compaction, > [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173] > has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from > earlier compaction. Probably because mob cleaner doesn't clean old hfiles. > Does this needs fixing? > 2. Across runs, same cell (ie. same key) may or may not be considered mob > reference cell. > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213]. > This at least happens with row keys 0 - 4 (which got compacted earlier). > [~jmhsieh] Any ideas why this would happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319924#comment-15319924 ] Hudson commented on HBASE-15948: SUCCESS: Integrated in HBase-Trunk_matrix #1008 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1008/]) Revert "HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns (stack: rev e66ecd7db68d6ef57084543d08f7774c82f22f45) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SimpleRpcSchedulerFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns (stack: rev 3a95552cfe6205ae845e1a7e1b5907da55b1a044) * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SimpleRpcSchedulerFactory.java > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15967) Metric for active ipc Readers and make default fraction of cpu count
[ https://issues.apache.org/jira/browse/HBASE-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319923#comment-15319923 ] Hudson commented on HBASE-15967: SUCCESS: Integrated in HBase-Trunk_matrix #1008 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1008/]) Revert "HBASE-15967 Metric for active ipc Readers and make default (stack: rev 6d5a25935e5ce983e14eff576a699ed1948566d2) * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java > Metric for active ipc Readers and make default fraction of cpu count > > > Key: HBASE-15967 > URL: https://issues.apache.org/jira/browse/HBASE-15967 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Attachments: HBASE-15967.master.001.patch > > > Our ipc Readers are hard coded at 10 regardless since . Running w/ less > Readers, we go faster..(e.g. 12 Readers has us doing 135k with workloadc and > 6 readers has us doing 145k).. .but hard to tell what count of Readers are > needed since no metric. > This issue changes Readers to be 1/4 the installed CPUs or 8, whichever is > the minimum, and then adds a new hbase.regionserver.ipc.runningReaders metric > so you have a chance seeing whats needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15968) Strong semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319918#comment-15319918
]
Duo Zhang commented on HBASE-15968:
---
We will use the sequence id of the HFile as the cell's sequence id if it is set
to 0. And I think we will not have multiples values for a same cell in one
HFile? I'm not sure, maybe we could still set sequence id to 0.
And as [~yangzhe1991] said above, we will introduce a table level config for
this feature. You can disable it if you do not want it.
Thanks.
> Strong semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15950) Fix memstore size estimates to be more tighter
[ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15950: -- Release Note: Made the object/array header sizes to be CompressedOops aware and fixed object heap size calculations for important objects like KeyValue and ConcurrentSkipListMap and . For heap sizes up to 32GB, depending on average cell sizes and total memstore size, %10-50 reduction in memstore size and flushes and compactions might be observed. YMMV. Due to more tighter than before size estimates, total heap space is expected to be utilized more, slightly increasing the chance to get OOM if misconfigured. Set the environment property (not hbase-site.xml) "hbase.memorylayout.use.unsafe" to false to disable. > Fix memstore size estimates to be more tighter > -- > > Key: HBASE-15950 > URL: https://issues.apache.org/jira/browse/HBASE-15950 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png, > hbase-15950-v0.patch > > > While testing something else, I was loading a region with a lot of data. > Writing 30M cells in 1M rows, with 1 byte values. > The memstore size turned out to be estimated as 4.5GB, while with the JFR > profiling I can see that we are using 2.8GB for all the objects in the > memstore (KV + KV byte[] + CSLM.Node + CSLM.Index). > This obviously means that there is room in the write cache that we are not > effectively using. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[ https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319905#comment-15319905 ] Jingcheng Du commented on HBASE-15959: -- bq. Across runs, same cell (ie. same key) may or may not be considered mob reference cell. here. This at least happens with row keys 0 - 4 (which got compacted earlier) Thanks a lot, Appy. I think this is design on purpose for this Test? In this test, it tries to verify the correctness that how mob cells are moved between store files and mob files. before the 1st compaction, the threshold is 0, it means all of the cells are considered as mob ones, and there are only reference cells in store files. In the 1st compaction, the threshold is changed to 100 which is larger than the value size of each mob cells. In this compaction, the cells are moved back to store files. In the scan after the 1st compaction, all the cells are the real ones, not reference ones. In the 2nd compaction, the threshold is changed to 0, and all cells are moved to mob files from store files. And in the second scan, the cells are the reference ones. > Fix flaky test TestRegionServerMetrics.testMobMetrics > - > > Key: HBASE-15959 > URL: https://issues.apache.org/jira/browse/HBASE-15959 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: huaxiang sun > > It flakes > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460]. > There are two weird things i identified: > 1. In second compaction, > [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173] > has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from > earlier compaction. Probably because mob cleaner doesn't clean old hfiles. > Does this needs fixing? > 2. Across runs, same cell (ie. same key) may or may not be considered mob > reference cell. > [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213]. > This at least happens with row keys 0 - 4 (which got compacted earlier). > [~jmhsieh] Any ideas why this would happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[
https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319890#comment-15319890
]
Jingcheng Du commented on HBASE-15959:
--
bq. In second compaction, scanner has 10 storefiles. Shouldn't there be 6? 5
from recent flushes and 1 from earlier compaction. Probably because mob cleaner
doesn't clean old hfiles. Does this needs fixing?
region.compact doesn't compact mob files, I think this is not related with
cleaners.
I read the test, I guess this issue is because the scanner is not closed in the
second scan. And the compactor cannot find the cells later than the smallest
read point?
{code}
scanner = table.getScanner(scan);
scanner.next(100);
numScanNext++;
+ scanner.close();
{code}
> Fix flaky test TestRegionServerMetrics.testMobMetrics
> -
>
> Key: HBASE-15959
> URL: https://issues.apache.org/jira/browse/HBASE-15959
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: huaxiang sun
>
> It flakes
> [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460].
> There are two weird things i identified:
> 1. In second compaction,
> [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173]
> has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from
> earlier compaction. Probably because mob cleaner doesn't clean old hfiles.
> Does this needs fixing?
> 2. Across runs, same cell (ie. same key) may or may not be considered mob
> reference cell.
> [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213].
> This at least happens with row keys 0 - 4 (which got compacted earlier).
> [~jmhsieh] Any ideas why this would happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-15466) precommit should not run all java goals when given a docs-only patch
[ https://issues.apache.org/jira/browse/HBASE-15466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey reassigned HBASE-15466: --- Assignee: Sean Busbey > precommit should not run all java goals when given a docs-only patch > > > Key: HBASE-15466 > URL: https://issues.apache.org/jira/browse/HBASE-15466 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Sean Busbey >Assignee: Sean Busbey > > Right now docs-only patches (those that only impact the top level > src/main/site, src/main/asciidoc, or src/main/xslt) run through all of the > java related precommit checks, including test4tests and the full unit test > suite. > Since we know these paths don't require those checks, we should update our > personality to skip them. (or fix our project structure to match "the maven > way".) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15985) clarify promises about edits from replication in ref guide
[ https://issues.apache.org/jira/browse/HBASE-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319889#comment-15319889 ] Sean Busbey commented on HBASE-15985: - this patch only touches the ref guide. both of the -1s are essentially HBASE-15466 > clarify promises about edits from replication in ref guide > -- > > Key: HBASE-15985 > URL: https://issues.apache.org/jira/browse/HBASE-15985 > Project: HBase > Issue Type: Sub-task > Components: documentation, Replication >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 2.0.0 > > Attachments: HBASE-15985.1.patch > > > we should make clear in a call out that replication only provides > at-least-once delivery and doesn't guarantee ordering so that e.g. folks > using increments aren't surprised. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15406) Split / merge switch left disabled after early termination of hbck
[
https://issues.apache.org/jira/browse/HBASE-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319886#comment-15319886
]
Heng Chen commented on HBASE-15406:
---
Agree. We should have one framework to do this things.
In current patch, we cleanup the lock and rollback the state by HBCK, so it
makes the logic much more complex than master do this work.
{quote}
One idea that he was proposing is to re-purpose the HBCK lock into being a
znode lock, and have the CJ, Balancer and Split/Merge look into whether HBCK is
running or not. I think this will be much simpler than adding 3 "advanced" APIs
to Admin. wdyt?
{quote}
Who will rollback the state and cleanup lock if hbck abort, Master or hbck ?
> Split / merge switch left disabled after early termination of hbck
> --
>
> Key: HBASE-15406
> URL: https://issues.apache.org/jira/browse/HBASE-15406
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Heng Chen
>Priority: Critical
> Labels: reviewed
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-15406.patch, HBASE-15406.v1.patch,
> HBASE-15406_v1.patch, HBASE-15406_v2.patch, test.patch, wip.patch
>
>
> This was what I did on cluster with 1.4.0-SNAPSHOT built Thursday:
> Run 'hbase hbck -disableSplitAndMerge' on gateway node of the cluster
> Terminate hbck early
> Enter hbase shell where I observed:
> {code}
> hbase(main):001:0> splitormerge_enabled 'SPLIT'
> false
> 0 row(s) in 0.3280 seconds
> hbase(main):002:0> splitormerge_enabled 'MERGE'
> false
> 0 row(s) in 0.0070 seconds
> {code}
> Expectation is that the split / merge switches should be restored to default
> value after hbck exits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15950) Fix memstore size estimates to be more tighter
[ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15950: -- Status: Patch Available (was: Open) > Fix memstore size estimates to be more tighter > -- > > Key: HBASE-15950 > URL: https://issues.apache.org/jira/browse/HBASE-15950 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png, > hbase-15950-v0.patch > > > While testing something else, I was loading a region with a lot of data. > Writing 30M cells in 1M rows, with 1 byte values. > The memstore size turned out to be estimated as 4.5GB, while with the JFR > profiling I can see that we are using 2.8GB for all the objects in the > memstore (KV + KV byte[] + CSLM.Node + CSLM.Index). > This obviously means that there is room in the write cache that we are not > effectively using. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15950) Fix memstore size estimates to be more tighter
[ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15950: -- Attachment: hbase-15950-v0.patch Here is an initial patch without JOL dependency. We use Unsafe to estimate the header sizes and oop reference size similar to the way JOL does it. For the changed ones, the estimates matches what JOL reports. Apart from small cell size tests where it gives >50% improvement in available memstore space, I've put this patch up in a YCSB cluster (single node). Inserted 30M rows of 1K size, with and without the patch with 50 regions. The number of flushes decreased from 610 to 500, and compactions from 64 to 41. Overall %20 decrease in flush / compaction IO is pretty significant. > Fix memstore size estimates to be more tighter > -- > > Key: HBASE-15950 > URL: https://issues.apache.org/jira/browse/HBASE-15950 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png, > hbase-15950-v0.patch > > > While testing something else, I was loading a region with a lot of data. > Writing 30M cells in 1M rows, with 1 byte values. > The memstore size turned out to be estimated as 4.5GB, while with the JFR > profiling I can see that we are using 2.8GB for all the objects in the > memstore (KV + KV byte[] + CSLM.Node + CSLM.Index). > This obviously means that there is room in the write cache that we are not > effectively using. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15950) Fix memstore size estimates to be more tighter
[ https://issues.apache.org/jira/browse/HBASE-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-15950: -- Summary: Fix memstore size estimates to be more tighter (was: We are grossly overestimating the memstore size) > Fix memstore size estimates to be more tighter > -- > > Key: HBASE-15950 > URL: https://issues.apache.org/jira/browse/HBASE-15950 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0 > > Attachments: Screen Shot 2016-06-02 at 8.48.27 PM.png > > > While testing something else, I was loading a region with a lot of data. > Writing 30M cells in 1M rows, with 1 byte values. > The memstore size turned out to be estimated as 4.5GB, while with the JFR > profiling I can see that we are using 2.8GB for all the objects in the > memstore (KV + KV byte[] + CSLM.Node + CSLM.Index). > This obviously means that there is room in the write cache that we are not > effectively using. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore
[
https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319877#comment-15319877
]
Heng Chen commented on HBASE-15900:
---
I have upgraded my hbase cluster hadoop version to be 2.6.0, it looks good now
on my production cluster (It has run three days.).
Let me run 2.6.0 at least one week before change hadoop version to be 2.7.2,
next week i will test it on my production cluster.
Wait for my good news! :)
> RS stuck in get lock of HStore
> --
>
> Key: HBASE-15900
> URL: https://issues.apache.org/jira/browse/HBASE-15900
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.1, 1.3.0
>Reporter: Heng Chen
> Attachments: 0d32a6bab354e6cc170cd59a2d485797.jstack.txt,
> 0d32a6bab354e6cc170cd59a2d485797.rs.log, 9fe15a52_9fe15a52_save,
> c91324eb_81194e359707acadee2906ffe36ab130.log, dump.txt
>
>
> It happens on my production cluster when i run MR job. I save the dump.txt
> from this RS webUI.
> Many threads stuck here:
> {code}
> Thread 133 (B.defaultRpcServer.handler=94,queue=4,port=16020):
>32 State: WAITING
>31 Blocked count: 477816
>30 Waited count: 535255
>29 Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6447ba67
>28 Stack:
>27 sun.misc.Unsafe.park(Native Method)
>26 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>25
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>24
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>23
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>22
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>21 org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:666)
>20
> org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:3621)
>19
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3038)
>18
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2793)
>17
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2735)
>16
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
>15
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
>14
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2029)
>13
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
>12 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
>11 org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>10
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> 9 org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> 8 java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15981) Stripe and Date-tiered compactions inaccurately suggest disabling table in docs
[
https://issues.apache.org/jira/browse/HBASE-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319863#comment-15319863
]
Enis Soztutar commented on HBASE-15981:
---
Remove hbase.online.schema.update.enable? Makes sense, it was a safeguard
config from long time ago. Mind creating issue?
> Stripe and Date-tiered compactions inaccurately suggest disabling table in
> docs
> ---
>
> Key: HBASE-15981
> URL: https://issues.apache.org/jira/browse/HBASE-15981
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15981.patch
>
>
> {{hbase.online.schema.update.enable}} is set to true in later versions of
> HBase. However, the documentation in the book around configuration Stripe and
> Date-tiered compactions instruct the user to disable a table before changing
> configuration. According to Enis on the user list, this is not necessary:
> {quote}
> Disabling the table should not be needed. From the stripe compaction
> perspective, deploying this in a disabled table versus in an online alter
> table is not different at all. The "hbase.online.schema.update.enable"
> property was fixing some possible race conditions that were fixed long time
> ago.
> {quote}
> We should update the book to remove this inaccurate messaging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15946) Eliminate possible security concerns in RS web UI's store file metrics
[
https://issues.apache.org/jira/browse/HBASE-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319865#comment-15319865
]
Enis Soztutar commented on HBASE-15946:
---
+1.
> Eliminate possible security concerns in RS web UI's store file metrics
> --
>
> Key: HBASE-15946
> URL: https://issues.apache.org/jira/browse/HBASE-15946
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Sean Mackrory
>Assignee: Mikhail Antonov
> Fix For: 1.3.0, 1.2.2
>
> Attachments: HBASE-15946-v1.patch, HBASE-15946-v2.patch,
> HBASE-15946-v3.patch
>
>
> More from static code analysis: it warns about the invoking of a separate
> command ("hbase hfile -s -f ...") as a possible security issue in
> hbase-server/src/main/resources/hbase-webapps/regionserver/storeFile.jsp.
> It looks to me like one cannot inject arbitrary shell script or even
> arbitrary arguments: ProcessBuilder makes that fairly safe and only allows
> the user to specify the argument that comes after -f. However that does
> potentially allow them to have the daemon's user access files they shouldn't
> be able to touch, albeit only for reading.
> To more explicitly eliminate any threats here, we should add some validation
> that the file is at least within HBase's root directory and use the Java API
> directly instead of invoking a separate executable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) Strong semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319861#comment-15319861
]
Phil Yang commented on HBASE-15968:
---
Yes, if we enable this feature, we will not remove the sequence id in HFile.
I'll post a design doc these days with more details. Thanks
> Strong semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15959) Fix flaky test TestRegionServerMetrics.testMobMetrics
[
https://issues.apache.org/jira/browse/HBASE-15959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319858#comment-15319858
]
huaxiang sun commented on HBASE-15959:
--
I have a patch ready. The root cause is
{code}
assertCounter("cellsCountCompactedToMob", 2 * numHfiles);
{code}
Assumes that there is no mob reference cells. Actually, after the first major
compact, the compacted files are still there in the family directory
(compaction.discharger does not get chance to kick in yet). When scan for the
next major compaction, these files may still be used, so scan may hit the cells
in these compacted files (hence it has mob reference).
The fix is to wait compaction.discharger to move compacted files to archive and
then start the second major compaction.
I will upload a patch after some clean up.
> Fix flaky test TestRegionServerMetrics.testMobMetrics
> -
>
> Key: HBASE-15959
> URL: https://issues.apache.org/jira/browse/HBASE-15959
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: huaxiang sun
>
> It flakes
> [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java#L460].
> There are two weird things i identified:
> 1. In second compaction,
> [scanner|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L173]
> has 10 storefiles. Shouldn't there be 6? 5 from recent flushes and 1 from
> earlier compaction. Probably because mob cleaner doesn't clean old hfiles.
> Does this needs fixing?
> 2. Across runs, same cell (ie. same key) may or may not be considered mob
> reference cell.
> [here|https://github.com/apache/hbase/blob/b557f0bec62a48753e5d01d7a47f3c9e5a6b3ee8/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java#L213].
> This at least happens with row keys 0 - 4 (which got compacted earlier).
> [~jmhsieh] Any ideas why this would happen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15988) Backup set add command MUST initiate full backup for a table(s) being added
Vladimir Rodionov created HBASE-15988: - Summary: Backup set add command MUST initiate full backup for a table(s) being added Key: HBASE-15988 URL: https://issues.apache.org/jira/browse/HBASE-15988 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore
[
https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319797#comment-15319797
]
Mikhail Antonov commented on HBASE-15900:
-
[~chenheng] curious if you had a chance to test your cluster on 2.7.2?
> RS stuck in get lock of HStore
> --
>
> Key: HBASE-15900
> URL: https://issues.apache.org/jira/browse/HBASE-15900
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.1, 1.3.0
>Reporter: Heng Chen
> Attachments: 0d32a6bab354e6cc170cd59a2d485797.jstack.txt,
> 0d32a6bab354e6cc170cd59a2d485797.rs.log, 9fe15a52_9fe15a52_save,
> c91324eb_81194e359707acadee2906ffe36ab130.log, dump.txt
>
>
> It happens on my production cluster when i run MR job. I save the dump.txt
> from this RS webUI.
> Many threads stuck here:
> {code}
> Thread 133 (B.defaultRpcServer.handler=94,queue=4,port=16020):
>32 State: WAITING
>31 Blocked count: 477816
>30 Waited count: 535255
>29 Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6447ba67
>28 Stack:
>27 sun.misc.Unsafe.park(Native Method)
>26 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>25
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>24
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>23
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>22
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>21 org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:666)
>20
> org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:3621)
>19
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3038)
>18
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2793)
>17
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2735)
>16
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
>15
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
>14
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2029)
>13
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
>12 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
>11 org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>10
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> 9 org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> 8 java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15985) clarify promises about edits from replication in ref guide
[
https://issues.apache.org/jira/browse/HBASE-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319794#comment-15319794
]
Hadoop QA commented on HBASE-15985:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 36s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.procedure.TestProcedureManager |
| | hadoop.hbase.master.balancer.TestRegionLocationFinder |
| | hadoop.hbase.mob.mapreduce.TestMobSweepMapper |
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestAsyncIPC |
| | org.apache.hadoop.hbase.ipc.TestIPC |
| | org.apache.hadoop.hbase.security.TestAsyncSecureIPC |
| | org.apache.hadoop.hbase.security.TestSecureIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808772/HBASE-15985.1.patch |
| JIRA Issue | HBASE-15985 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 2da090f |
| Default Java | 1.7.0_79 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/2142/artifact/patchprocess/patch-unit-root.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/2142/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/2142/testReport/ |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2142/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> clarify promises about edits from replication in ref guide
> --
>
> Key: HBASE-15985
> URL: https://issues.apache.org/jira/browse/HBASE-15985
> Project: HBase
> Issue Type: Sub-task
> Components: documentation, Replication
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 2.0.0
>
> Attachments: HBASE-15985.1.patch
>
>
> we should make clear in a call out that replication only provides
> at-least-once delivery and doesn't guarantee ordering so that e.g. folks
>
[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore
[
https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319795#comment-15319795
]
Mikhail Antonov commented on HBASE-15900:
-
For 1.3 I think 2.7.2 should be the default version, for earlier release lines
we may not be able to change the defaults but may recommend newer versions.
> RS stuck in get lock of HStore
> --
>
> Key: HBASE-15900
> URL: https://issues.apache.org/jira/browse/HBASE-15900
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.1, 1.3.0
>Reporter: Heng Chen
> Attachments: 0d32a6bab354e6cc170cd59a2d485797.jstack.txt,
> 0d32a6bab354e6cc170cd59a2d485797.rs.log, 9fe15a52_9fe15a52_save,
> c91324eb_81194e359707acadee2906ffe36ab130.log, dump.txt
>
>
> It happens on my production cluster when i run MR job. I save the dump.txt
> from this RS webUI.
> Many threads stuck here:
> {code}
> Thread 133 (B.defaultRpcServer.handler=94,queue=4,port=16020):
>32 State: WAITING
>31 Blocked count: 477816
>30 Waited count: 535255
>29 Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6447ba67
>28 Stack:
>27 sun.misc.Unsafe.park(Native Method)
>26 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>25
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>24
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>23
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>22
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>21 org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:666)
>20
> org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:3621)
>19
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3038)
>18
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2793)
>17
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2735)
>16
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
>15
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
>14
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2029)
>13
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
>12 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
>11 org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>10
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> 9 org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> 8 java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15900) RS stuck in get lock of HStore
[
https://issues.apache.org/jira/browse/HBASE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319790#comment-15319790
]
Andrew Purtell commented on HBASE-15900:
We also saw long stuck flushes or compactions, and blocked region actions,
until we patched our Hadoop for 7005. There have been other fixes for missing
timeouts committed since. We probably should be recommending 2.7.2 or 2.6.4 for
production.
> RS stuck in get lock of HStore
> --
>
> Key: HBASE-15900
> URL: https://issues.apache.org/jira/browse/HBASE-15900
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.1, 1.3.0
>Reporter: Heng Chen
> Attachments: 0d32a6bab354e6cc170cd59a2d485797.jstack.txt,
> 0d32a6bab354e6cc170cd59a2d485797.rs.log, 9fe15a52_9fe15a52_save,
> c91324eb_81194e359707acadee2906ffe36ab130.log, dump.txt
>
>
> It happens on my production cluster when i run MR job. I save the dump.txt
> from this RS webUI.
> Many threads stuck here:
> {code}
> Thread 133 (B.defaultRpcServer.handler=94,queue=4,port=16020):
>32 State: WAITING
>31 Blocked count: 477816
>30 Waited count: 535255
>29 Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6447ba67
>28 Stack:
>27 sun.misc.Unsafe.park(Native Method)
>26 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>25
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>24
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>23
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>22
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>21 org.apache.hadoop.hbase.regionserver.HStore.add(HStore.java:666)
>20
> org.apache.hadoop.hbase.regionserver.HRegion.applyFamilyMapToMemstore(HRegion.java:3621)
>19
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3038)
>18
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2793)
>17
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2735)
>16
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:692)
>15
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:654)
>14
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2029)
>13
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
>12 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2112)
>11 org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
>10
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> 9 org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> 8 java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13887) Document 0.98 release build differences
[
https://issues.apache.org/jira/browse/HBASE-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319787#comment-15319787
]
Andrew Purtell commented on HBASE-13887:
The instructions are slightly out of date. The BouncyCastle provider release
versions linked above 404. Find the latest at
https://www.bouncycastle.org/latest_releases.html .
> Document 0.98 release build differences
> ---
>
> Key: HBASE-13887
> URL: https://issues.apache.org/jira/browse/HBASE-13887
> Project: HBase
> Issue Type: Bug
> Components: documentation, site
>Reporter: Andrew Purtell
> Fix For: 2.0.0
>
>
> The build instructions in the online manual do not describe the extra steps
> we need to take to build 0.98. Add a section on this.
> A quick enumeration of the differences:
> 1. Source assemblies will be missing the hbase-hadoop1-compat module. This
> should be fixed in the POM somehow. What I do now is untar the src tarball,
> cp -a the module over, then tar up the result. (It's a hack in a release
> script.)
> 2. We must munge POMs for building hadoop1 and hadoop2 variants and then
> execute two builds pointing Maven at each munged POM. The
> generate-hadoop-X-poms script requires bash
> {noformat}
> $ bash dev-support/generate-hadoopX-poms.sh $version $version-hadoop1
> $ bash dev-support/generate-hadoopX-poms.sh $version $version-hadoop2
> {noformat}
> Build Hadoop 1
> {noformat}
> $ mvn -f pom.xml.hadoop1 clean install -DskipTests -Prelease && \
> mvn -f pom.xml.hadoop1 install -DskipTests site assembly:single \
> -Prelease && \
> mvn -f pom.xml.hadoop1 deploy -DskipTests -Papache-release
> $ cp hbase-assembly/target/hbase*-bin.tar.gz $release_dir
> {noformat}
> Build Hadoop 2
> {noformat}
> $ mvn -f pom.xml.hadoop2 clean install -DskipTests -Prelease && \
> mvn -f pom.xml.hadoop2 install -DskipTests site assembly:single \
> -Prelease && \
> mvn -f pom.xml.hadoop2 deploy -DskipTests -Papache-release
> $ cp hbase-assembly/target/hbase*-bin.tar.gz $release_dir
> {noformat}
> 3. Current HEAD of 0.98 branch enforces a requirement that the release be
> built with a JDK no more recent than the compile language level. For 0.98,
> that is 1.6, therefore the ancient 6u45 JDK. This JDK suffers from
> [JDK-6521495|http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6521495] so
> the following workaround is required in order to deploy artifacts to Apache's
> Nexus:
> 3.a. Download https://www.bouncycastle.org/download/bcprov-jdk15on-152.jar
> and https://www.bouncycastle.org/download/bcprov-ext-jdk15on-152.jar into
> $JAVA_HOME/lib/ext.
> 3.b. Edit $JAVA_HOME/lib/security/java.security and add the BouncyCastle
> provider as the first provider:
> {noformat}
> security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15986) BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
[
https://issues.apache.org/jira/browse/HBASE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-15986:
--
Attachment: HBASE-15986-v2.patch
v2. Catching Exception instead of IOException.
> BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
> --
>
> Key: HBASE-15986
> URL: https://issues.apache.org/jira/browse/HBASE-15986
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: HBASE-15986-v1.patch, HBASE-15986-v2.patch
>
>
> I was running TestBackupDescribe where I saw the following being passed to
> BackupServerUtil.getWALFilesOlderThan():
> hdfs://localhost:59278/user/tyu/test-data/a42ac21a-2097-49d8-9c0e-86991e104e4e/oldWALs/hregion-05273083.default.146587177
> Since the filename doesn't represent a server, the following call throws
> exception:
> {code}
> ServerName.parsePort(String) line: 150
> BackupClientUtil.parseHostFromOldLog(Path) line: 136
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15986) BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
[
https://issues.apache.org/jira/browse/HBASE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319770#comment-15319770
]
Vladimir Rodionov commented on HBASE-15986:
---
No, couple more places: parseHostNameFromLogFile in BackupServerUtil, for
example.
> BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
> --
>
> Key: HBASE-15986
> URL: https://issues.apache.org/jira/browse/HBASE-15986
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: HBASE-15986-v1.patch
>
>
> I was running TestBackupDescribe where I saw the following being passed to
> BackupServerUtil.getWALFilesOlderThan():
> hdfs://localhost:59278/user/tyu/test-data/a42ac21a-2097-49d8-9c0e-86991e104e4e/oldWALs/hregion-05273083.default.146587177
> Since the filename doesn't represent a server, the following call throws
> exception:
> {code}
> ServerName.parsePort(String) line: 150
> BackupClientUtil.parseHostFromOldLog(Path) line: 136
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319768#comment-15319768
]
Hudson commented on HBASE-15957:
SUCCESS: Integrated in HBase-1.1-JDK7 #1728 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1728/])
HBASE-15957 RpcClientImpl.close never ends in some circumstances (enis: rev
4418ba2f8f10ee08d82aee9f311e0e1b6bd7be5e)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/ipc/IntegrationTestRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6
>
> Attachments: HBASE-15957-2.patch, HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15987) [0.98] Downgrade maven-site-plugin to 2.4
[ https://issues.apache.org/jira/browse/HBASE-15987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15987: --- Attachment: HBASE-15987-0.98.patch > [0.98] Downgrade maven-site-plugin to 2.4 > - > > Key: HBASE-15987 > URL: https://issues.apache.org/jira/browse/HBASE-15987 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Blocker > Fix For: 0.98.20 > > Attachments: HBASE-15987-0.98.patch > > > Downgrade to maven-site-plugin version 2.4 > > We needed to find a version of the site plugin that would not pull in a > plugin dependency compiled with Java > 1.6. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15987) [0.98] Downgrade maven-site-plugin to 2.4
Andrew Purtell created HBASE-15987: -- Summary: [0.98] Downgrade maven-site-plugin to 2.4 Key: HBASE-15987 URL: https://issues.apache.org/jira/browse/HBASE-15987 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Blocker Fix For: 0.98.20 Downgrade to maven-site-plugin version 2.4 We needed to find a version of the site plugin that would not pull in a plugin dependency compiled with Java > 1.6. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15987) [0.98] Downgrade maven-site-plugin to 2.4
[ https://issues.apache.org/jira/browse/HBASE-15987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-15987. Resolution: Fixed > [0.98] Downgrade maven-site-plugin to 2.4 > - > > Key: HBASE-15987 > URL: https://issues.apache.org/jira/browse/HBASE-15987 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Blocker > Fix For: 0.98.20 > > > Downgrade to maven-site-plugin version 2.4 > > We needed to find a version of the site plugin that would not pull in a > plugin dependency compiled with Java > 1.6. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319729#comment-15319729
]
Hudson commented on HBASE-15957:
SUCCESS: Integrated in HBase-1.1-JDK8 #1815 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1815/])
HBASE-15957 RpcClientImpl.close never ends in some circumstances (enis: rev
4418ba2f8f10ee08d82aee9f311e0e1b6bd7be5e)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/ipc/IntegrationTestRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6
>
> Attachments: HBASE-15957-2.patch, HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15946) Eliminate possible security concerns in RS web UI's store file metrics
[
https://issues.apache.org/jira/browse/HBASE-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319727#comment-15319727
]
Mikhail Antonov commented on HBASE-15946:
-
[~enis] want to have another look?
> Eliminate possible security concerns in RS web UI's store file metrics
> --
>
> Key: HBASE-15946
> URL: https://issues.apache.org/jira/browse/HBASE-15946
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Sean Mackrory
>Assignee: Mikhail Antonov
> Fix For: 1.3.0, 1.2.2
>
> Attachments: HBASE-15946-v1.patch, HBASE-15946-v2.patch,
> HBASE-15946-v3.patch
>
>
> More from static code analysis: it warns about the invoking of a separate
> command ("hbase hfile -s -f ...") as a possible security issue in
> hbase-server/src/main/resources/hbase-webapps/regionserver/storeFile.jsp.
> It looks to me like one cannot inject arbitrary shell script or even
> arbitrary arguments: ProcessBuilder makes that fairly safe and only allows
> the user to specify the argument that comes after -f. However that does
> potentially allow them to have the daemon's user access files they shouldn't
> be able to touch, albeit only for reading.
> To more explicitly eliminate any threats here, we should add some validation
> that the file is at least within HBase's root directory and use the Java API
> directly instead of invoking a separate executable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15946) Eliminate possible security concerns in RS web UI's store file metrics
[
https://issues.apache.org/jira/browse/HBASE-15946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319725#comment-15319725
]
Mikhail Antonov commented on HBASE-15946:
-
[~mackrorysd] yeah, that's exactly what I meant - glad it worked.
Looked at the last patch, +1. Thanks for picking it up!
> Eliminate possible security concerns in RS web UI's store file metrics
> --
>
> Key: HBASE-15946
> URL: https://issues.apache.org/jira/browse/HBASE-15946
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.2.1
>Reporter: Sean Mackrory
>Assignee: Mikhail Antonov
> Fix For: 1.3.0, 1.2.2
>
> Attachments: HBASE-15946-v1.patch, HBASE-15946-v2.patch,
> HBASE-15946-v3.patch
>
>
> More from static code analysis: it warns about the invoking of a separate
> command ("hbase hfile -s -f ...") as a possible security issue in
> hbase-server/src/main/resources/hbase-webapps/regionserver/storeFile.jsp.
> It looks to me like one cannot inject arbitrary shell script or even
> arbitrary arguments: ProcessBuilder makes that fairly safe and only allows
> the user to specify the argument that comes after -f. However that does
> potentially allow them to have the daemon's user access files they shouldn't
> be able to touch, albeit only for reading.
> To more explicitly eliminate any threats here, we should add some validation
> that the file is at least within HBase's root directory and use the Java API
> directly instead of invoking a separate executable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15800) check_compatibility.sh broken as of upstream b8f125f
[
https://issues.apache.org/jira/browse/HBASE-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319703#comment-15319703
]
Nick Dimiduk commented on HBASE-15800:
--
Looks like I never opened an upstream issue about this. Nor did i follow up on
the other [upstream
issue|https://github.com/lvc/japi-compliance-checker/issues/23] I did open...
> check_compatibility.sh broken as of upstream b8f125f
>
>
> Key: HBASE-15800
> URL: https://issues.apache.org/jira/browse/HBASE-15800
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Previously (recently as 2 weeks ago) it was possible to use the
> {{dev-support/check_compatibility.sh}} script against the tags under {{rel}}.
> Now ({{5e0}}) this is no longer the case. On a mac,
> {noformat}
> $ echo $JAVA_HOME
> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
> $ GETOPT=/usr/local/Cellar/gnu-getopt/1.1.6/bin/getopt
> ./dev-support/check_compatibility.sh -r
> https://git-wip-us.apache.org/repos/asf/hbase.git rel/1.1.4 branch-1.1
> ...
> Running the Java API Compliance Checker...
> ERROR: can't access
> './dev-support/target/compatibility/1/hbase-annotations/target/hbase-annotations-1.1.4.jar,./dev-support/target/compatibility/1/hbase-checkstyle/target/hbase-checkstyle-1.1.
> 4.jar,./dev-support/target/compatibility/1/hbase-client/target/hbase-client-1.1.4.jar,./dev-support/target/compatibility/1/hbase-common/target/hbase-common-1.1.4.jar,./dev-support/target/compat
> ibility/1/hbase-examples/target/hbase-examples-1.1.4.jar,./dev-support/target/compatibility/1/hbase-hadoop-compat/target/hbase-hadoop-compat-1.1.4.jar,./dev-support/target/compatibility/1/hbase
> -hadoop2-compat/target/hbase-hadoop2-compat-1.1.4.jar,./dev-support/target/compatibility/1/hbase-it/target/hbase-it-1.1.4.jar,./dev-support/target/compatibility/1/hbase-prefix-tree/target/hbase
> -prefix-tree-1.1.4.jar,./dev-support/target/compatibility/1/hbase-procedure/target/hbase-procedure-1.1.4.jar,./dev-support/target/compatibility/1/hbase-protocol/target/hbase-protocol-1.1.4.jar,
> ./dev-support/target/compatibility/1/hbase-resource-bundle/target/hbase-resource-bundle-1.1.4.jar,./dev-support/target/compatibility/1/hbase-rest/target/hbase-rest-1.1.4.jar,./dev-support/targe
> t/compatibility/1/hbase-server/target/hbase-server-1.1.4.jar,./dev-support/target/compatibility/1/hbase-shell/target/hbase-shell-1.1.4.jar,./dev-support/target/compatibility/1/hbase-testing-uti
> l/target/hbase-testing-util-1.1.4.jar,./dev-support/target/compatibility/1/hbase-thrift/target/hbase-thrift-1.1.4.jar'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15948: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Would take a bit of work going back to branch-1. > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319698#comment-15319698 ] stack commented on HBASE-15948: --- Ok. Reverted what was there in case it was wrong version and then applied 004. Thanks for the review [~enis] > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319692#comment-15319692 ] stack commented on HBASE-15948: --- Hmm.. mistaken commit. I'll leave it in but I miscommitted other stuff too... Let me revert that. Thanks for the review [~enis] > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15986) BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
[
https://issues.apache.org/jira/browse/HBASE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319686#comment-15319686
]
Ted Yu commented on HBASE-15986:
The catch is only needed in parseHostFromOldLog(), right ?
> BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
> --
>
> Key: HBASE-15986
> URL: https://issues.apache.org/jira/browse/HBASE-15986
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: HBASE-15986-v1.patch
>
>
> I was running TestBackupDescribe where I saw the following being passed to
> BackupServerUtil.getWALFilesOlderThan():
> hdfs://localhost:59278/user/tyu/test-data/a42ac21a-2097-49d8-9c0e-86991e104e4e/oldWALs/hregion-05273083.default.146587177
> Since the filename doesn't represent a server, the following call throws
> exception:
> {code}
> ServerName.parsePort(String) line: 150
> BackupClientUtil.parseHostFromOldLog(Path) line: 136
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15986) BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
[
https://issues.apache.org/jira/browse/HBASE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-15986:
--
Attachment: HBASE-15986-v1.patch
patch cc: [~tedyu]
> BackupServerUtil.getWALFilesOlderThan() should handle oldWALs properly
> --
>
> Key: HBASE-15986
> URL: https://issues.apache.org/jira/browse/HBASE-15986
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: HBASE-15986-v1.patch
>
>
> I was running TestBackupDescribe where I saw the following being passed to
> BackupServerUtil.getWALFilesOlderThan():
> hdfs://localhost:59278/user/tyu/test-data/a42ac21a-2097-49d8-9c0e-86991e104e4e/oldWALs/hregion-05273083.default.146587177
> Since the filename doesn't represent a server, the following call throws
> exception:
> {code}
> ServerName.parsePort(String) line: 150
> BackupClientUtil.parseHostFromOldLog(Path) line: 136
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319672#comment-15319672 ] Enis Soztutar commented on HBASE-15948: --- Already committed? Anyway belated +1. > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[
https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319670#comment-15319670
]
Hadoop QA commented on HBASE-15130:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-15130 does not apply to 0.98. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12792592/HBASE-15130-0.98.v4.patch
|
| JIRA Issue | HBASE-15130 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/2143/console |
| Powered by | Apache Yetus 0.2.1 http://yetus.apache.org |
This message was automatically generated.
> Backport 0.98 Scan different TimeRange for each column family
> --
>
> Key: HBASE-15130
> URL: https://issues.apache.org/jira/browse/HBASE-15130
> Project: HBase
> Issue Type: Bug
> Components: Client, regionserver, Scanners
>Affects Versions: 0.98.17
>Reporter: churro morales
>Assignee: churro morales
> Fix For: 0.98.21
>
> Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch,
> HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch,
> HBASE-15130-0.98.v3.patch, HBASE-15130-0.98.v4.patch
>
>
> branch 98 version backport for HBASE-14355
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15957) RpcClientImpl.close never ends in some circumstances
[
https://issues.apache.org/jira/browse/HBASE-15957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319659#comment-15319659
]
Hudson commented on HBASE-15957:
FAILURE: Integrated in HBase-Trunk_matrix #1007 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1007/])
HBASE-15957 RpcClientImpl.close never ends in some circumstances (enis: rev
da88b4824054f57fbcbc7795469ab2369a39b5ed)
*
hbase-it/src/test/java/org/apache/hadoop/hbase/ipc/IntegrationTestRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java
> RpcClientImpl.close never ends in some circumstances
>
>
> Key: HBASE-15957
> URL: https://issues.apache.org/jira/browse/HBASE-15957
> Project: HBase
> Issue Type: Bug
> Components: Client, rpc
>Affects Versions: 1.1.2
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 1.1.6
>
> Attachments: HBASE-15957-2.patch, HBASE-15957.patch
>
>
> This bug is related to HBASE-14241 and HBASE-13851.
> Fix for HBASE-13851 introduced the check for non alive connections and if
> connection is not alive, it close it:
> {noformat}
> if (!conn.isAlive()) {
> if (connsToClose == null) {
> connsToClose = new HashSet();
> }
> connsToClose.add(conn);
> }
>
> if (connsToClose != null) {
> for (Connection conn : connsToClose) {
> if (conn.markClosed(new InterruptedIOException("RpcClient is
> closing"))) {
> conn.close();
> }
> }
> }
> {noformat}
> That worked fine until fix for HBASE-14241 introduced handling for interrupt
> in writer thread:
> {noformat}
> try {
> cts = callsToWrite.take();
> } catch (InterruptedException e) {
> markClosed(new InterruptedIOException());
> }
> {noformat}
> So, if writer thread is running, but connection thread is not started yet,
> interrupt will cause calling of markClosed which will set
> shouldCloseConnection flag for the parent connection. And the next time
> during the handling of non alive connections markClosed will return false and
> close will not be called. As the result connection will not be removed from
> the connections pool and RpcClientImpl.close never finish.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319658#comment-15319658 ] Hudson commented on HBASE-15948: FAILURE: Integrated in HBase-Trunk_matrix #1007 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1007/]) HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns (stack: rev e0b70c00e74aeaac33570508e3732a53daea839e) * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/AbstractTestIPC.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SimpleRpcSchedulerFactory.java > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15981) Stripe and Date-tiered compactions inaccurately suggest disabling table in docs
[
https://issues.apache.org/jira/browse/HBASE-15981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319656#comment-15319656
]
Hudson commented on HBASE-15981:
FAILURE: Integrated in HBase-Trunk_matrix #1007 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1007/])
HBASE-15981 Remove references to disabling table in docs around stripe (stack:
rev 2da090f9a3ca42b07f342035c952ce0465c67265)
* src/main/asciidoc/_chapters/architecture.adoc
> Stripe and Date-tiered compactions inaccurately suggest disabling table in
> docs
> ---
>
> Key: HBASE-15981
> URL: https://issues.apache.org/jira/browse/HBASE-15981
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15981.patch
>
>
> {{hbase.online.schema.update.enable}} is set to true in later versions of
> HBase. However, the documentation in the book around configuration Stripe and
> Date-tiered compactions instruct the user to disable a table before changing
> configuration. According to Enis on the user list, this is not necessary:
> {quote}
> Disabling the table should not be needed. From the stripe compaction
> perspective, deploying this in a disabled table versus in an online alter
> table is not different at all. The "hbase.online.schema.update.enable"
> property was fixing some possible race conditions that were fixed long time
> ago.
> {quote}
> We should update the book to remove this inaccurate messaging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15967) Metric for active ipc Readers and make default fraction of cpu count
[ https://issues.apache.org/jira/browse/HBASE-15967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319657#comment-15319657 ] Hudson commented on HBASE-15967: FAILURE: Integrated in HBase-Trunk_matrix #1007 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1007/]) HBASE-15967 Metric for active ipc Readers and make default fraction of (stack: rev 1125215aad3f5b149f3458ba7019c5920f6dca66) * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Metric for active ipc Readers and make default fraction of cpu count > > > Key: HBASE-15967 > URL: https://issues.apache.org/jira/browse/HBASE-15967 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: stack > Attachments: HBASE-15967.master.001.patch > > > Our ipc Readers are hard coded at 10 regardless since . Running w/ less > Readers, we go faster..(e.g. 12 Readers has us doing 135k with workloadc and > 6 readers has us doing 145k).. .but hard to tell what count of Readers are > needed since no metric. > This issue changes Readers to be 1/4 the installed CPUs or 8, whichever is > the minimum, and then adds a new hbase.regionserver.ipc.runningReaders metric > so you have a chance seeing whats needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15258) Clean up javadoc warnings and errors on 0.98 branch
[ https://issues.apache.org/jira/browse/HBASE-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15258: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Clean up javadoc warnings and errors on 0.98 branch > --- > > Key: HBASE-15258 > URL: https://issues.apache.org/jira/browse/HBASE-15258 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.98.21 > > > Clean up javadoc warnings and errors on 0.98 branch so issues reported in > precommit build results are more meaningful and actionable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13096) NPE from SecureWALCellCodec$EncryptedKvEncoder#write when using WAL encryption and Phoenix secondary indexes
[
https://issues.apache.org/jira/browse/HBASE-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-13096:
---
Fix Version/s: (was: 0.98.20)
0.98.21
> NPE from SecureWALCellCodec$EncryptedKvEncoder#write when using WAL
> encryption and Phoenix secondary indexes
>
>
> Key: HBASE-13096
> URL: https://issues.apache.org/jira/browse/HBASE-13096
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.6
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Labels: phoenix
> Fix For: 0.98.21
>
>
> On user@phoenix Dhavi Rami reported:
> {quote}
> I tried using phoenix in hBase with Transparent Encryption of Data At Rest
> enabled ( AES encryption)
> Works fine for a table with primary key column.
> But it doesn't work if I create Secondary index on that tables.I tried to dig
> deep into the problem and found WAL file encryption throws exception when I
> have Global Secondary Index created on my mutable table.
> Following is the error I was getting on one of the region server.
> {noformat}
> 2015-02-20 10:44:48,768 ERROR
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: UNEXPECTED
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:767)
> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:754)
> at org.apache.hadoop.hbase.KeyValue.getKeyLength(KeyValue.java:1253)
> at
> org.apache.hadoop.hbase.regionserver.wal.SecureWALCellCodec$EncryptedKvEncoder.write(SecureWALCellCodec.java:194)
> at
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:117)
> at
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1137)
> at java.lang.Thread.run(Thread.java:745)
> 2015-02-20 10:44:48,776 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog:
> regionserver60020-WAL.AsyncWriter exiting
> {noformat}
> I had to disable WAL encryption, and it started working fine with secondary
> Index. So Hfile encryption works with secondary index but WAL encryption
> doesn't work.
> {quote}
> Parking this here for later investigation. For now I'm going to assume this
> is something in SecureWALCellCodec that needs looking at, but if it turns out
> to be a Phoenix indexer issue I will move this JIRA there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11290) Unlock RegionStates
[ https://issues.apache.org/jira/browse/HBASE-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11290: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Unlock RegionStates > --- > > Key: HBASE-11290 > URL: https://issues.apache.org/jira/browse/HBASE-11290 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.2.0, 1.3.0 >Reporter: Francis Liu >Assignee: Francis Liu > Fix For: 2.0.0, 1.4.0, 0.98.21 > > Attachments: HBASE-11290-0.98.patch, HBASE-11290-0.98_v2.patch, > HBASE-11290.01.branch-1.patch, HBASE-11290.02.patch, HBASE-11290.03.patch, > HBASE-11290.draft.patch, HBASE-11290_trunk.patch > > > Even though RegionStates is a highly accessed data structure in HMaster. Most > of it's methods are synchronized. Which limits concurrency. Even simply > making some of the getters non-synchronized by using concurrent data > structures has helped with region assignments. We can go as simple as this > approach or create locks per region or a bucket lock per region bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14546) Backport stub DNS re-resolution options to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14546: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport stub DNS re-resolution options to 0.98 > --- > > Key: HBASE-14546 > URL: https://issues.apache.org/jira/browse/HBASE-14546 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.98.21 > > > HBASE-12943 and HBASE-13067 addresses infinite caching preventing servers > from rejoining a cluster using the same hostname but a different IP address. > HBASE-14544 modifies this to be optional. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15257) Clean up findbugs warnings on 0.98 branch
[ https://issues.apache.org/jira/browse/HBASE-15257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15257: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Clean up findbugs warnings on 0.98 branch > - > > Key: HBASE-15257 > URL: https://issues.apache.org/jira/browse/HBASE-15257 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.98.21 > > > Findbugs reports from precommit builds against 0.98 branch will be much more > useful if we make a pass over the existing significant warnings and fix them, > and add annotations for warnings we expect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14177) Full GC on client may lead to missing scan results
[ https://issues.apache.org/jira/browse/HBASE-14177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14177: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Full GC on client may lead to missing scan results > -- > > Key: HBASE-14177 > URL: https://issues.apache.org/jira/browse/HBASE-14177 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.12, 0.98.13, 1.0.2 >Reporter: James Estes >Priority: Critical > Labels: dataloss > Fix For: 0.98.21 > > > After adding a large row, scanning back that row winds up being empty. After > a few attempts it will succeed (all attempts over the same data on an hbase > getting no other writes). > Looking at logs, it seems this happens when there is memory pressure on the > client and there are several Full GCs that happen. Then messages that > indicate that region locations are being removed from the local client cache: > 2015-07-31 12:50:24,647 [main] DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation > - Removed 192.168.1.131:50981 as a location of > big_row_1438368609944,,1438368610048.880c849594807bdc7412f4f982337d6c. for > tableName=big_row_1438368609944 from cache > Blaming the GC may sound fanciful, but if the test is run with -Xms4g -Xmx4g > then it always passes on the first scan attempt. Maybe the pause is enough to > remove something from the cache, or the client is using weak references > somewhere? > More info > http://mail-archives.apache.org/mod_mbox/hbase-user/201507.mbox/%3CCAE8tVdnFf%3Dob569%3DfJkpw1ndVWOVTkihYj9eo6qt0FrzihYHgw%40mail.gmail.com%3E > Test used to reproduce: > https://github.com/housejester/hbase-debugging#fullgctest > I tested and had failures in: > 0.98.12 client/server > 0.98.13 client 0.98.12 server > 0.98.13 client/server > 1.1.0 client 0.98.13 server > 0.98.13 client and 1.1.0 server > 0.98.12 client and 1.1.0 server > I tested without failure in: > 1.1.0 client/server -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15924) Enhance hbase services autorestart capability to hbase-daemon.sh
[ https://issues.apache.org/jira/browse/HBASE-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15924: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Enhance hbase services autorestart capability to hbase-daemon.sh > - > > Key: HBASE-15924 > URL: https://issues.apache.org/jira/browse/HBASE-15924 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.19 >Reporter: Loknath Priyatham Teja Singamsetty >Assignee: Loknath Priyatham Teja Singamsetty > Fix For: 0.98.21 > > > As part of HBASE-5939, the autorestart for hbase services has been added to > deal with scenarios where hbase services (master/regionserver/master-backup) > gets killed or goes down leading to unplanned outages. The changes were made > to hbase-daemon.sh to support autorestart option. > However, the autorestart implementation doesn't work in standalone mode and > other than that have few gaps with the implementation as per release notes of > HBASE-5939. Here is an attempt to re-design and fix the functionality > considering all possible usecases with hbase service operations. > Release Notes of HBASE-5939: > -- > When launched with autorestart, HBase processes will automatically restart if > they are not properly terminated, either by a "stop" command or by a cluster > stop. To ensure that it does not overload the system when the server itself > is corrupted and the process cannot be restarted, the server sleeps for 5 > minutes before restarting if it was already started 5 minutes ago previously. > To use it, launch the process with "bin/start-hbase autorestart". This option > is not fully compatible with the existing "restart" command: if you ask for a > restart on a server launched with autorestart, the server will restart but > the next server instance won't be automatically restarted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12148: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.21 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen > Shot 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > Screen Shot 2016-04-13 at 1.49.30 PM.png, Screen Shot 2016-04-13 at 2.02.22 > PM.png, Screen Shot 2016-05-18 at 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14932) bulkload fails because file not found
[ https://issues.apache.org/jira/browse/HBASE-14932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14932: --- Fix Version/s: (was: 0.98.20) 0.98.21 > bulkload fails because file not found > - > > Key: HBASE-14932 > URL: https://issues.apache.org/jira/browse/HBASE-14932 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.10 >Reporter: Shuaifeng Zhou > Fix For: 0.98.21 > > Attachments: HBASE-14932-0.98.patch > > > When make a dobulkload call, one call may contain sevel hfiles to load, but > the call may timeout during regionserver load files, and client will retry to > load. > But when client doing retry call, regionserver may continue doing load > operation, if somefiles success, the retry call will throw filenotfound > exception, and this will cause client retry again and again until retry > exhausted, and bulkload fails. > When this happening, actually, some files are loaded successfully, that's a > inconsistent status. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13511) Derive data keys with HKDF
[ https://issues.apache.org/jira/browse/HBASE-13511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13511: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Derive data keys with HKDF > -- > > Key: HBASE-13511 > URL: https://issues.apache.org/jira/browse/HBASE-13511 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 0.98.21 > > > When we are locally managing master key material, when users have supplied > their own data key material, derive the actual data keys using HKDF > (https://tools.ietf.org/html/rfc5869) > DK' = HKDF(S, DK, MK) > where > S = salt > DK = user supplied data key > MK = master key > DK' = derived data key for the HFile > User supplied key material may be weak or an attacker may have some partial > knowledge of it. > Where we generate random data keys we can still use HKDF as a way to mix more > entropy into the secure random generator. > DK' = HKDF(R, MK) > where > R = random key material drawn from the system's secure random generator > MK = master key > (Salting isn't useful here because salt S and R would be drawn from the same > pool, so will not have statistical independence.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14945) Backport HBASE-13153 (Bulk loaded HFile replication) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14945: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport HBASE-13153 (Bulk loaded HFile replication) to 0.98 > > > Key: HBASE-14945 > URL: https://issues.apache.org/jira/browse/HBASE-14945 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Purtell > Fix For: 0.98.21 > > > HBASE-13153 (Bulk loaded HFile replication) looks like a good candidate to > backport to 0.98. It would address an important functional gap in replication > (bulk loads don't replicate) for 0.98 users, is a new feature toggled with a > new configuration parameter that is by default off, and does not change any > interfaces not marked as Private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15130) Backport 0.98 Scan different TimeRange for each column family
[ https://issues.apache.org/jira/browse/HBASE-15130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15130: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport 0.98 Scan different TimeRange for each column family > -- > > Key: HBASE-15130 > URL: https://issues.apache.org/jira/browse/HBASE-15130 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 0.98.17 >Reporter: churro morales >Assignee: churro morales > Fix For: 0.98.21 > > Attachments: HBASE-15130-0.98.patch, HBASE-15130-0.98.v1.patch, > HBASE-15130-0.98.v1.patch, HBASE-15130-0.98.v2.patch, > HBASE-15130-0.98.v3.patch, HBASE-15130-0.98.v4.patch > > > branch 98 version backport for HBASE-14355 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13506) AES-GCM cipher support where available
[ https://issues.apache.org/jira/browse/HBASE-13506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13506: --- Fix Version/s: (was: 0.98.20) 0.98.21 > AES-GCM cipher support where available > -- > > Key: HBASE-13506 > URL: https://issues.apache.org/jira/browse/HBASE-13506 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.3.0, 0.98.21 > > > The initial encryption drop only had AES-CTR support because authenticated > modes such as GCM are only available in Java 7 and up, and our trunk at the > time was targeted at Java 6. However we can optionally use AES-GCM cipher > support where available. For HBase 1.0 and up, Java 7 is now the minimum so > use of AES-GCM can go in directly. It's probably possible to add support in > 0.98 too using reflection for cipher object initialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14049) SnapshotHFileCleaner should optionally clean up after failed snapshots
[ https://issues.apache.org/jira/browse/HBASE-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14049: --- Fix Version/s: (was: 0.98.20) 0.98.21 > SnapshotHFileCleaner should optionally clean up after failed snapshots > -- > > Key: HBASE-14049 > URL: https://issues.apache.org/jira/browse/HBASE-14049 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.13 >Reporter: Andrew Purtell > Fix For: 2.0.0, 1.3.0, 0.98.21 > > > SnapshotHFileCleaner should optionally clean up after failed snapshots rather > than just complain. Add a configuration option that, if set to true > (defaulting to false), instructs SnapshotHFileCleaner to recursively remove > failed snapshot temporary directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15454) Archive store files older than max age
[ https://issues.apache.org/jira/browse/HBASE-15454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15454: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Archive store files older than max age > -- > > Key: HBASE-15454 > URL: https://issues.apache.org/jira/browse/HBASE-15454 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Affects Versions: 2.0.0, 1.3.0, 0.98.18, 1.4.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0, 1.4.0, 0.98.21 > > Attachments: HBASE-15454-v1.patch, HBASE-15454-v2.patch, > HBASE-15454-v3.patch, HBASE-15454-v4.patch, HBASE-15454-v5.patch, > HBASE-15454-v6.patch, HBASE-15454-v7.patch, HBASE-15454.patch > > > In date tiered compaction, the store files older than max age are never > touched by minor compactions. Here we introduce a 'freeze window' operation, > which does the follow things: > 1. Find all store files that contains cells whose timestamp are in the give > window. > 2. Compaction all these files and output one file for each window that these > files covered. > After the compaction, we will have only one in the give window, and all cells > whose timestamp are in the give window are in the only file. And if you do > not write new cells with an older timestamp in this window, the file will > never be changed. This makes it easier to do erasure coding on the freezed > file to reduce redundence. And also, it makes it possible to check > consistency between master and peer cluster incrementally. > And why use the word 'freeze'? > Because there is already an 'HFileArchiver' class. I want to use a different > word to prevent confusing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15580) Tag coprocessor limitedprivate scope to StoreFile.Reader
[
https://issues.apache.org/jira/browse/HBASE-15580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15580:
---
Fix Version/s: (was: 0.98.20)
0.98.21
> Tag coprocessor limitedprivate scope to StoreFile.Reader
>
>
> Key: HBASE-15580
> URL: https://issues.apache.org/jira/browse/HBASE-15580
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Fix For: 2.0.0, 1.0.4, 1.2.2, 1.1.6, 0.98.21
>
> Attachments: HBASE-15580.patch, HBASE-15580_branch-1.0.patch
>
>
> For phoenix local indexing we need to have custom storefile reader
> constructor(IndexHalfStoreFileReader) to distinguish from other storefile
> readers. So wanted to mark StoreFile.Reader scope as
> InterfaceAudience.LimitedPrivate("Coprocessor")
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14289) Backport HBASE-13965 'Stochastic Load Balancer JMX Metrics' to 0.98
[ https://issues.apache.org/jira/browse/HBASE-14289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14289: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport HBASE-13965 'Stochastic Load Balancer JMX Metrics' to 0.98 > --- > > Key: HBASE-14289 > URL: https://issues.apache.org/jira/browse/HBASE-14289 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.21 > > Attachments: 14289-0.98-v2.txt, 14289-0.98-v3.txt, 14289-0.98-v4.txt, > 14289-0.98-v5.txt > > > The default HBase load balancer (the Stochastic load balancer) is cost > function based. The cost function weights are tunable but no visibility into > those cost function results is directly provided. > This issue backports HBASE-13965 to 0.98 branch to provide visibility via JMX > into each cost function of the stochastic load balancer, as well as the > overall cost of the balancing plan. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15635) Mean age of Blocks in cache (seconds) on webUI should be greater than zero
[ https://issues.apache.org/jira/browse/HBASE-15635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15635: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Mean age of Blocks in cache (seconds) on webUI should be greater than zero > -- > > Key: HBASE-15635 > URL: https://issues.apache.org/jira/browse/HBASE-15635 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen >Assignee: Heng Chen > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 1.0.5, 1.1.6, 0.98.21 > > Attachments: 7BFFAF68-0807-400C-853F-706B498449E1.png, > HBASE-15635.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13504) Alias current AES cipher as AES-CTR
[ https://issues.apache.org/jira/browse/HBASE-13504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13504: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Alias current AES cipher as AES-CTR > --- > > Key: HBASE-13504 > URL: https://issues.apache.org/jira/browse/HBASE-13504 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 0.98.21 > > > Alias the current cipher with the name "AES" to the name "AES-CTR". -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14276) NPE when setting up stub for connection to master if secure connect is refused
[
https://issues.apache.org/jira/browse/HBASE-14276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14276:
---
Fix Version/s: (was: 0.98.20)
0.98.21
> NPE when setting up stub for connection to master if secure connect is refused
> --
>
> Key: HBASE-14276
> URL: https://issues.apache.org/jira/browse/HBASE-14276
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.21
>
>
> If an insecure client contacts a secure cluster we can get an NPE when
> setting up the stub for the master connection. We should throw an IOException
> with a clear message, and not retry if possible to distinguish this case.
> Found in 0.98 but might be relevant for later branches
> {noformat}
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
> INFO: Process identifier=hconnection-0x3c1e23ff connecting to ZooKeeper
> ensemble=x.x.x.x:2181,x.x.x.x:2181,x.x.x.x:2181
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> makeStub
> INFO: getMaster attempt 1 of 35 failed; retrying after sleep of 100,
> exception=com.google.protobuf.ServiceException: java.lang.NullPointerException
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> makeStub
> INFO: getMaster attempt 2 of 35 failed; retrying after sleep of 200,
> exception=com.google.protobuf.ServiceException: java.io.IOException: Call to
> /x.x.x.x:6 failed on local exception: java.io.EOFException
> Aug 20, 2015 12:04:31 AM
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
> makeStub
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14792) Latest docs fail to build when packaging 0.98
[
https://issues.apache.org/jira/browse/HBASE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-14792:
---
Fix Version/s: (was: 0.98.20)
0.98.21
> Latest docs fail to build when packaging 0.98
> -
>
> Key: HBASE-14792
> URL: https://issues.apache.org/jira/browse/HBASE-14792
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 0.98.21
>
> Attachments: HBASE-14792-0.98.patch
>
>
> For releasing 0.98 we typically copy back the latest docs from master and
> then build.
> When I try generating the latest docs, I get:
> {noformat}
> [INFO]
>
> [INFO] Forking Apache HBase - Assembly 0.98.16-hadoop2
> [INFO]
>
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.0.1:enforce (enforce) @ hbase-assembly ---
> [INFO]
> [INFO] --- buildnumber-maven-plugin:1.3:create-timestamp (default) @
> hbase-assembly ---
> [INFO]
> [INFO] <<< maven-javadoc-plugin:2.9.1:aggregate (report:aggregate) <
> generate-sources @ hbase <<<
> [INFO] configuring report plugin
> org.apache.maven.plugins:maven-checkstyle-plugin:2.13
> [INFO] Parent project loaded from repository: org.apache:apache:pom:12
> [INFO] Relativizing decoration links with respect to project URL:
> http://hbase.apache.org
> [INFO] Rendering site with org.apache.maven.skins:maven-fluido-skin:jar:1.4
> skin.
> [INFO] Skipped "About" report, file "index.html" already exists for the
> English version.
> [INFO] Skipped "Source Xref" report, file "xref/index.html" already exists
> for the English version.
> [INFO] Skipped "Test Source Xref" report, file "xref-test/index.html" already
> exists for the English version.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
> hbase: Error during page generation: Could not find the template
> 'META-INF/maven/site.vm: Encountered "source" at META-INF/maven/site.vm[line
> 1162, column 52]
> [ERROR] Was expecting one of:
> [ERROR] "," ...
> [ERROR] ")" ...
> [ERROR] ...
> {noformat}
> Do I need to make POM updates?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14845) hbase-server leaks jdk.tools dependency to mapreduce consumers
[ https://issues.apache.org/jira/browse/HBASE-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14845: --- Fix Version/s: (was: 0.98.20) 0.98.21 > hbase-server leaks jdk.tools dependency to mapreduce consumers > -- > > Key: HBASE-14845 > URL: https://issues.apache.org/jira/browse/HBASE-14845 > Project: HBase > Issue Type: Bug > Components: build, dependencies >Affects Versions: 2.0.0, 0.98.14, 1.2.0, 1.1.2, 1.3.0, 1.0.3 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.2.2, 1.1.6, 1.3.1, 0.98.21 > > Attachments: HBASE-14845.1.patch > > > HBASE-13963 / HBASE-14844 take care of removing leaks of our dependency on > jdk-tools. > Until we move the mapreduce support classes out of hbase-server > (HBASE-11843), we need to also avoid leaking the dependency from that module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14259) Backport Namespace quota support to 98 branch
[ https://issues.apache.org/jira/browse/HBASE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14259: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport Namespace quota support to 98 branch > -- > > Key: HBASE-14259 > URL: https://issues.apache.org/jira/browse/HBASE-14259 > Project: HBase > Issue Type: Task >Reporter: Vandana Ayyalasomayajula >Assignee: Andrew Purtell > Fix For: 0.98.21 > > Attachments: HBASE-14259_v1_0.98.patch, HBASE-14259_v2_0.98.patch, > HBASE-14259_v3_0.98.patch > > > Namespace quota support (HBASE-8410) has been backported to branch-1 > (HBASE-13438). This jira would backport the same to 98 branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15984: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.0.4, 1.4.0, 1.2.2, 1.1.6, 0.98.21 > > Attachments: HBASE-15984.1.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14927) Backport HBASE-13014 and HBASE-14749 to branch-1 and 0.98
[ https://issues.apache.org/jira/browse/HBASE-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-14927: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport HBASE-13014 and HBASE-14749 to branch-1 and 0.98 > - > > Key: HBASE-14927 > URL: https://issues.apache.org/jira/browse/HBASE-14927 > Project: HBase > Issue Type: Improvement >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan > Fix For: 1.3.0, 0.98.21 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13667) Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks
[ https://issues.apache.org/jira/browse/HBASE-13667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13667: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Backport HBASE-12975 to 1.0 and 0.98 without changing coprocessors hooks > > > Key: HBASE-13667 > URL: https://issues.apache.org/jira/browse/HBASE-13667 > Project: HBase > Issue Type: Bug >Reporter: Rajeshbabu Chintaguntla >Assignee: Rajeshbabu Chintaguntla > Fix For: 0.98.21 > > > We can backport Split transaction, region merge transaction interfaces to > branch 1.0 and 0.98 without changing coprocessor hooks. Then it should be > compatible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15775) Canary launches two AuthUtil Chores
[ https://issues.apache.org/jira/browse/HBASE-15775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15775: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Canary launches two AuthUtil Chores > --- > > Key: HBASE-15775 > URL: https://issues.apache.org/jira/browse/HBASE-15775 > Project: HBase > Issue Type: Bug > Components: canary >Affects Versions: 0.98.13, 0.98.14, 1.2.0, 0.98.15, 1.2.1, 0.98.16, > 0.98.17, 0.98.18, 0.98.16.1, 0.98.19 >Reporter: Sean Busbey >Assignee: Vishal Khandelwal >Priority: Minor > Labels: beginner > Fix For: 1.2.2, 0.98.21 > > Attachments: HBASE-15775.0.98.00.patch > > > Looks like a result of an error in backport done in HBASE-13712. We have a > AuthUtil chore both in main() and in run(). > The one in main() should be removed so that the code is consistent with other > branches. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13505) Deprecate the "AES" cipher type
[ https://issues.apache.org/jira/browse/HBASE-13505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13505: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Deprecate the "AES" cipher type > --- > > Key: HBASE-13505 > URL: https://issues.apache.org/jira/browse/HBASE-13505 > Project: HBase > Issue Type: Sub-task > Components: encryption, security >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0, 1.3.0, 0.98.21 > > > Deprecate the "AES" cipher type. Remove internal references to it and use the > "AES-CTR" name instead -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13336) Consistent rules for security meta table protections
[ https://issues.apache.org/jira/browse/HBASE-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-13336: --- Fix Version/s: (was: 0.98.20) 0.98.21 > Consistent rules for security meta table protections > > > Key: HBASE-13336 > URL: https://issues.apache.org/jira/browse/HBASE-13336 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov > Fix For: 2.0.0, 1.3.0, 0.98.21 > > Attachments: HBASE-13336.patch, HBASE-13336_v2.patch > > > The AccessController and VisibilityController do different things regarding > protecting their meta tables. The AC allows schema changes and disable/enable > if the user has permission. The VC unconditionally disallows all admin > actions. Generally, bad things will happen if these meta tables are damaged, > disabled, or dropped. The likely outcome is random frequent (or constant) > server side op failures with nasty stack traces. On the other hand some > things like column family and table attribute changes can have valid use > cases. We should have consistent and sensible rules for protecting security > meta tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15600) Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks
[
https://issues.apache.org/jira/browse/HBASE-15600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-15600:
---
Fix Version/s: (was: 0.98.20)
0.98.21
> Add provision for adding mutations to memstore or able to write to same
> region in batchMutate coprocessor hooks
> ---
>
> Key: HBASE-15600
> URL: https://issues.apache.org/jira/browse/HBASE-15600
> Project: HBase
> Issue Type: Improvement
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
> Labels: phoenix
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21
>
> Attachments: HBASE-15600.patch, HBASE-15600_v1.patch,
> HBASE-15600_v2.patch, hbase-15600_v3.patch, hbase-15600_v4.patch,
> hbase-15600_v5.patch, hbase-15600_v6.patch
>
>
> As part of PHOENIX-1734 we need to write the index updates to same region
> from coprocessors but writing from batchMutate API is not allowed because of
> mvcc.
> Raised PHOENIX-2742 to discuss any alternative way to write to the same
> region directly or not but not having any proper solution there.
> Currently we have provision to write wal edits from coprocessors. We can set
> wal edits in MiniBatchOperationInProgress.
> {noformat}
> /**
>* Sets the walEdit for the operation(Mutation) at the specified position.
>* @param index
>* @param walEdit
>*/
> public void setWalEdit(int index, WALEdit walEdit) {
> this.walEditsFromCoprocessors[getAbsoluteIndex(index)] = walEdit;
> }
> {noformat}
> Similarly we can allow to write mutations from coprocessors to memstore as
> well. Or else we should provide the batch mutation API allow write in batch
> mutate coprocessors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12148) Remove TimeRangeTracker as point of contention when many threads writing a Store
[ https://issues.apache.org/jira/browse/HBASE-12148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12148: --- Fix Version/s: (was: 0.98.19) 0.98.20 > Remove TimeRangeTracker as point of contention when many threads writing a > Store > > > Key: HBASE-12148 > URL: https://issues.apache.org/jira/browse/HBASE-12148 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0, 0.99.1 >Reporter: stack >Assignee: Walter Koetke > Fix For: 2.0.0, 1.3.0, 0.98.20 > > Attachments: > 0001-In-AtomicUtils-change-updateMin-and-updateMax-to-ret.patch, > 12148.addendum.txt, 12148.min_and_max_run_independent.patch, 12148.txt, > 12148.txt, 12148v2.txt, 12148v2.txt, 12148v4.patch, HBASE-12148-V3.patch, > HBASE-12148-V3.patch, HBASE-12148.branch-1.v5.patch, > HBASE-12148.branch-1.v5.patch, HBASE-12148.txt, HBASE-12148V2.txt, Screen > Shot 2014-10-01 at 3.39.46 PM.png, Screen Shot 2014-10-01 at 3.41.07 PM.png, > Screen Shot 2016-04-13 at 1.49.30 PM.png, Screen Shot 2016-04-13 at 2.02.22 > PM.png, Screen Shot 2016-05-18 at 10.21.53 PM.png, TimeRangeTracker.tiff > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319650#comment-15319650 ] Enis Soztutar commented on HBASE-15948: --- Checking. > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15948) Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
[ https://issues.apache.org/jira/browse/HBASE-15948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319647#comment-15319647 ] stack commented on HBASE-15948: --- Can I have a +1 on version 004 please? > Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" > - > > Key: HBASE-15948 > URL: https://issues.apache.org/jira/browse/HBASE-15948 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-15948.master.001.patch, > HBASE-15948.master.002.patch, HBASE-15948.master.003.patch, > HBASE-15948.master.004.patch, HBASE-15948.master.004.patch, > HBASE-15948.master.005.patch, Screen Shot 2016-06-02 at 6.16.39 PM.png > > > Esteban noticed we were missing this upstream issue. Seems to make no > difference in profiling but here is the patch anyways. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[
https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319643#comment-15319643
]
Hadoop QA commented on HBASE-15984:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 33s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 39s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 111m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mob.mapreduce.TestMobSweepMapper |
| | hadoop.hbase.procedure.TestProcedureManager |
| | hadoop.hbase.master.balancer.TestRegionLocationFinder |
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestAsyncIPC |
| | org.apache.hadoop.hbase.ipc.TestIPC |
| | org.apache.hadoop.hbase.security.TestAsyncSecureIPC |
| | org.apache.hadoop.hbase.security.TestSecureIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808759/HBASE-15984.1.patch |
| JIRA Issue | HBASE-15984 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf907.gq1.ygridcore.net
[jira] [Commented] (HBASE-15800) check_compatibility.sh broken as of upstream b8f125f
[
https://issues.apache.org/jira/browse/HBASE-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319633#comment-15319633
]
Andrew Purtell commented on HBASE-15800:
Broken on Linux as well (Ubuntu 16.04 LTS)
> check_compatibility.sh broken as of upstream b8f125f
>
>
> Key: HBASE-15800
> URL: https://issues.apache.org/jira/browse/HBASE-15800
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0
>Reporter: Nick Dimiduk
>Priority: Minor
>
> Previously (recently as 2 weeks ago) it was possible to use the
> {{dev-support/check_compatibility.sh}} script against the tags under {{rel}}.
> Now ({{5e0}}) this is no longer the case. On a mac,
> {noformat}
> $ echo $JAVA_HOME
> /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
> $ GETOPT=/usr/local/Cellar/gnu-getopt/1.1.6/bin/getopt
> ./dev-support/check_compatibility.sh -r
> https://git-wip-us.apache.org/repos/asf/hbase.git rel/1.1.4 branch-1.1
> ...
> Running the Java API Compliance Checker...
> ERROR: can't access
> './dev-support/target/compatibility/1/hbase-annotations/target/hbase-annotations-1.1.4.jar,./dev-support/target/compatibility/1/hbase-checkstyle/target/hbase-checkstyle-1.1.
> 4.jar,./dev-support/target/compatibility/1/hbase-client/target/hbase-client-1.1.4.jar,./dev-support/target/compatibility/1/hbase-common/target/hbase-common-1.1.4.jar,./dev-support/target/compat
> ibility/1/hbase-examples/target/hbase-examples-1.1.4.jar,./dev-support/target/compatibility/1/hbase-hadoop-compat/target/hbase-hadoop-compat-1.1.4.jar,./dev-support/target/compatibility/1/hbase
> -hadoop2-compat/target/hbase-hadoop2-compat-1.1.4.jar,./dev-support/target/compatibility/1/hbase-it/target/hbase-it-1.1.4.jar,./dev-support/target/compatibility/1/hbase-prefix-tree/target/hbase
> -prefix-tree-1.1.4.jar,./dev-support/target/compatibility/1/hbase-procedure/target/hbase-procedure-1.1.4.jar,./dev-support/target/compatibility/1/hbase-protocol/target/hbase-protocol-1.1.4.jar,
> ./dev-support/target/compatibility/1/hbase-resource-bundle/target/hbase-resource-bundle-1.1.4.jar,./dev-support/target/compatibility/1/hbase-rest/target/hbase-rest-1.1.4.jar,./dev-support/targe
> t/compatibility/1/hbase-server/target/hbase-server-1.1.4.jar,./dev-support/target/compatibility/1/hbase-shell/target/hbase-shell-1.1.4.jar,./dev-support/target/compatibility/1/hbase-testing-uti
> l/target/hbase-testing-util-1.1.4.jar,./dev-support/target/compatibility/1/hbase-thrift/target/hbase-thrift-1.1.4.jar'
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
