[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[
https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505681#comment-15505681
]
Hadoop QA commented on HBASE-16264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m 51s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 160 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 56s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 34m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
8s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 47s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 56s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 45s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 45s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 45s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 38m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
50s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2298 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 11s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
41m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s
{color} | {color:green} root generated 0 new + 19 unchanged - 1 fixed = 19
total (was 20) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} hbase-client generated 0 new + 13 unchanged - 1 fixed =
13 total (was 14) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 12s
{color} | {color:red} hbase-procedure generated 6 new + 0 unchanged - 0 fixed =
6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 7s
{color}
[jira] [Updated] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16648: -- Status: Open (was: Patch Available) > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505666#comment-15505666 ] Duo Zhang commented on HBASE-16648: --- Write a simple JMH test, it shows that the default implementation of computeIfAbsent in ConcurrentMap interface is faster than the version of ConcurrentHashMap when there is no contention. We need to figure out why before commit this patch. Thanks [~ikeda], big help here. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505655#comment-15505655 ] Hiroshi Ikeda commented on HBASE-16648: --- My bad, I didn't realized that the default implementation is overridden by the interface. Although CopyOnWriteArrayMap.putIfAbsent is wrongly implemented, it seems computeIfAbsent works well. bq. Lock does not reduce performance, the problem is contention. Generally speaking, lock can reduce performance by context switches when contention occurs, while CAS doesn't block. Moreover the implementation is not so trivial and causes overhead to ensure atomicity in ConcurretHashMap. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[ https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505639#comment-15505639 ] Allan Yang commented on HBASE-16649: I mean add the line 'flushedSequenceIdByRegion.put(encodedRegionName, l);' In trunk, when smaller seq id is reported, it will be ignored. My point is that is no harm to update the seqid even if it is smaller. Otherwise, if this situation do happen, data loss it is a sure thing. > Truncate table with splits preserved can cause both data loss and truncated > data appeared again > --- > > Key: HBASE-16649 > URL: https://issues.apache.org/jira/browse/HBASE-16649 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.3 >Reporter: Allan Yang >Assignee: Matteo Bertozzi > Attachments: HBASE-16649-v0.patch, HBASE-16649-v1.patch > > > Since truncate table with splits preserved will delete hfiles and use the > previous regioninfo. It can cause odd behaviors > - Case 1: *Data appeared after truncate* > reproduce procedure： > 1. create a table, let's say 'test' > 2. write data to 'test', make sure memstore of 'test' is not empty > 3. truncate 'test' with splits preserved > 4. kill the regionserver hosting the region(s) of 'test' > 5. start the regionserver, now it is the time to witness the miracle! the > truncated data appeared in table 'test' > - Case 2: *Data loss* > reproduce procedure: > 1. create a table, let's say 'test' > 2. write some data to 'test', no matter how many > 3. truncate 'test' with splits preserved > 4. restart the regionserver to reset the seqid > 5. write some data, but less than 2 since we don't want the seqid to run over > the one in 2 > 6. kill the regionserver hosting the region(s) of 'test' > 7. restart the regionserver. Congratulations! the data writen in 4 is now all > lost > *Why?* > for case 1 > Since preserve splits in truncate table procedure will not change the > regioninfo, when log replay happens, the 'unflushed' data will restore back > to the region > for case 2 > since the flushedSequenceIdByRegion are stored in Master in a map with the > region's encodedName. Although the table is truncated, the region's name is > not changed since we chose to preserve the splits. So after truncate the > table, the region's sequenceid is reset in the regionserver, but not reset in > master. When flush comes and report to master, master will reject the update > of sequenceid since the new one is smaller than the old one. The same happens > in log replay, all the edits writen in 4 will be skipped since they have a > smaller seqid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[
https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505595#comment-15505595
]
Hadoop QA commented on HBASE-16648:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 35s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 22s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 6s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterShutdown |
| Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsckOneRS |
| | org.apache.hadoop.hbase.client.TestAdmin1 |
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| |
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829318/HBASE-16648-v2.patch |
| JIRA Issue | HBASE-16648 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 73d36639d3ce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014
[jira] [Commented] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution
[
https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505562#comment-15505562
]
Hadoop QA commented on HBASE-16587:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 36m 6s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 45s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 114m 59s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 210m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
| | hadoop.hbase.client.TestAdmin1 |
| | hadoop.hbase.regionserver.TestRegionServerMetrics |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout |
| | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829309/HBASE-16587-v3.patch |
| JIRA Issue | HBASE-16587 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 247781294dc1 3.13.0-92-generic #139-Ubuntu SMP Tue
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[
https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505548#comment-15505548
]
Hadoop QA commented on HBASE-16649:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 42s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 9s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
| |
org.apache.hadoop.hbase.master.normalizer.TestSimpleRegionNormalizerOnCluster |
| | org.apache.hadoop.hbase.master.TestRollingRestart |
| | org.apache.hadoop.hbase.master.TestDistributedLogSplitting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829316/HBASE-16649-v1.patch |
| JIRA Issue | HBASE-16649 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 24fe8c64ed9b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2eac0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3614/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3614/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3614/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3614/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This
[jira] [Comment Edited] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505542#comment-15505542 ] chunhui shen edited comment on HBASE-16630 at 9/20/16 4:34 AM: --- 1.Fix the issue that FreeSpace doesn't free anything (Required) 2.Defragmentation (Optional): a. Would increment cache usage ratio in short time. Otherwise will take relative long time without defragmentation. b. Resource overhead (Byte Copy). Should limit the rate. For example, only trigger one time per hour. c. Add unit test to ensure the correctness with defragmentation. Just my thought. was (Author: zjushch): 1.Fix the issue that FreeSpace doesn't free anything (Required) 2.Defragmentation (Optional): a. Would increment cache usage ratio in short time. Otherwise will take relative long time without defragmentation. b. Resource overhead (Byte Copy). Should limit the rate. For example, only trigger one time per hour. c. Add unit test to ensure the correctness with defragmentation. Just my thought. optional > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505542#comment-15505542 ] chunhui shen commented on HBASE-16630: -- 1.Fix the issue that FreeSpace doesn't free anything (Required) 2.Defragmentation (Optional): a. Would increment cache usage ratio in short time. Otherwise will take relative long time without defragmentation. b. Resource overhead (Byte Copy). Should limit the rate. For example, only trigger one time per hour. c. Add unit test to ensure the correctness with defragmentation. Just my thought. optional > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505478#comment-15505478
]
Hadoop QA commented on HBASE-16257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 38s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 53s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 14s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829314/HBASE-16257-v5.patch |
| JIRA Issue | HBASE-16257 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux 2912f63a6ac9 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
[jira] [Updated] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-16648: -- Attachment: HBASE-16648-v2.patch Remove the modification of WeakObjectPool as [~ikeda] suggested, we need more tests here. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, > HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505360#comment-15505360 ] Duo Zhang commented on HBASE-16648: --- Lock does not reduce performance, the problem is contention. And I think we could modify WeakObjectPool in another jira, where we could run several tests to see the performance impact. And I'd say the default implementation in ConcurrentMap is not wrong. It is exactly what you always do without a computeIfAbsent method. Thanks. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-7612) [JDK8] Replace use of high-scale-lib counters with intrinsic facilities
[ https://issues.apache.org/jira/browse/HBASE-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505346#comment-15505346 ] Hiroshi Ikeda commented on HBASE-7612: -- Instead of reset, using another AtomicLonger to keep the last sum and calculating the difference will help you to avoid dropping increments. > [JDK8] Replace use of high-scale-lib counters with intrinsic facilities > --- > > Key: HBASE-7612 > URL: https://issues.apache.org/jira/browse/HBASE-7612 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Duo Zhang >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-7612-v1.patch, HBASE-7612.patch > > > JEP155 introduces a few new classes (DoubleAccumulator, DoubleAdder, > LongAccumulator, LongAdder) that "internally employ contention-reduction > techniques that provide huge throughput improvements as compared to Atomic > variables". There are applications of these where we are currently using > Cliff Click's high-scale-lib and for metrics. > See http://openjdk.java.net/jeps/155 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent
[ https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505335#comment-15505335 ] Hiroshi Ikeda commented on HBASE-16648: --- I read just halfway the patch, but behaviors of Map.computeIfAbsent depend on their implementations. For example, CopyOnWriteArrayMap defined in HBase uses the default implementation, which means MetaCache in the patch is wrong at least. ConcurrentHashMap uses an exclusive lock and has possibility to reduce performance. The contract of WeakObjectPool is changed with the possibility in the patch and you should do something about that. > [JDK8] Use computeIfAbsent instead of get and putIfAbsent > - > > Key: HBASE-16648 > URL: https://issues.apache.org/jira/browse/HBASE-16648 > Project: HBase > Issue Type: Sub-task > Components: Performance >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16648-v1.patch, HBASE-16648.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[ https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16649: Attachment: HBASE-16649-v1.patch > Truncate table with splits preserved can cause both data loss and truncated > data appeared again > --- > > Key: HBASE-16649 > URL: https://issues.apache.org/jira/browse/HBASE-16649 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.3 >Reporter: Allan Yang >Assignee: Matteo Bertozzi > Attachments: HBASE-16649-v0.patch, HBASE-16649-v1.patch > > > Since truncate table with splits preserved will delete hfiles and use the > previous regioninfo. It can cause odd behaviors > - Case 1: *Data appeared after truncate* > reproduce procedure： > 1. create a table, let's say 'test' > 2. write data to 'test', make sure memstore of 'test' is not empty > 3. truncate 'test' with splits preserved > 4. kill the regionserver hosting the region(s) of 'test' > 5. start the regionserver, now it is the time to witness the miracle! the > truncated data appeared in table 'test' > - Case 2: *Data loss* > reproduce procedure: > 1. create a table, let's say 'test' > 2. write some data to 'test', no matter how many > 3. truncate 'test' with splits preserved > 4. restart the regionserver to reset the seqid > 5. write some data, but less than 2 since we don't want the seqid to run over > the one in 2 > 6. kill the regionserver hosting the region(s) of 'test' > 7. restart the regionserver. Congratulations! the data writen in 4 is now all > lost > *Why?* > for case 1 > Since preserve splits in truncate table procedure will not change the > regioninfo, when log replay happens, the 'unflushed' data will restore back > to the region > for case 2 > since the flushedSequenceIdByRegion are stored in Master in a map with the > region's encodedName. Although the table is truncated, the region's name is > not changed since we chose to preserve the splits. So after truncate the > table, the region's sequenceid is reset in the regionserver, but not reset in > master. When flush comes and report to master, master will reject the update > of sequenceid since the new one is smaller than the old one. The same happens > in log replay, all the edits writen in 4 will be skipped since they have a > smaller seqid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[ https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505298#comment-15505298 ] Duo Zhang commented on HBASE-16649: --- Theoretically this should not happen? I mean a smaller flush sequence id. And remove this can not solve all the problems as you may do log replay before the new sequence id reported. > Truncate table with splits preserved can cause both data loss and truncated > data appeared again > --- > > Key: HBASE-16649 > URL: https://issues.apache.org/jira/browse/HBASE-16649 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.3 >Reporter: Allan Yang >Assignee: Matteo Bertozzi > Attachments: HBASE-16649-v0.patch > > > Since truncate table with splits preserved will delete hfiles and use the > previous regioninfo. It can cause odd behaviors > - Case 1: *Data appeared after truncate* > reproduce procedure： > 1. create a table, let's say 'test' > 2. write data to 'test', make sure memstore of 'test' is not empty > 3. truncate 'test' with splits preserved > 4. kill the regionserver hosting the region(s) of 'test' > 5. start the regionserver, now it is the time to witness the miracle! the > truncated data appeared in table 'test' > - Case 2: *Data loss* > reproduce procedure: > 1. create a table, let's say 'test' > 2. write some data to 'test', no matter how many > 3. truncate 'test' with splits preserved > 4. restart the regionserver to reset the seqid > 5. write some data, but less than 2 since we don't want the seqid to run over > the one in 2 > 6. kill the regionserver hosting the region(s) of 'test' > 7. restart the regionserver. Congratulations! the data writen in 4 is now all > lost > *Why?* > for case 1 > Since preserve splits in truncate table procedure will not change the > regioninfo, when log replay happens, the 'unflushed' data will restore back > to the region > for case 2 > since the flushedSequenceIdByRegion are stored in Master in a map with the > region's encodedName. Although the table is truncated, the region's name is > not changed since we chose to preserve the splits. So after truncate the > table, the region's sequenceid is reset in the regionserver, but not reset in > master. When flush comes and report to master, master will reject the update > of sequenceid since the new one is smaller than the old one. The same happens > in log replay, all the edits writen in 4 will be skipped since they have a > smaller seqid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[
https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505293#comment-15505293
]
Duo Zhang commented on HBASE-16649:
---
{code}
private static List recreteRegionInfo(final List
regions) {
{code}
Typo? recreate?
> Truncate table with splits preserved can cause both data loss and truncated
> data appeared again
> ---
>
> Key: HBASE-16649
> URL: https://issues.apache.org/jira/browse/HBASE-16649
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.3
>Reporter: Allan Yang
>Assignee: Matteo Bertozzi
> Attachments: HBASE-16649-v0.patch
>
>
> Since truncate table with splits preserved will delete hfiles and use the
> previous regioninfo. It can cause odd behaviors
> - Case 1: *Data appeared after truncate*
> reproduce procedure：
> 1. create a table, let's say 'test'
> 2. write data to 'test', make sure memstore of 'test' is not empty
> 3. truncate 'test' with splits preserved
> 4. kill the regionserver hosting the region(s) of 'test'
> 5. start the regionserver, now it is the time to witness the miracle! the
> truncated data appeared in table 'test'
> - Case 2: *Data loss*
> reproduce procedure:
> 1. create a table, let's say 'test'
> 2. write some data to 'test', no matter how many
> 3. truncate 'test' with splits preserved
> 4. restart the regionserver to reset the seqid
> 5. write some data, but less than 2 since we don't want the seqid to run over
> the one in 2
> 6. kill the regionserver hosting the region(s) of 'test'
> 7. restart the regionserver. Congratulations! the data writen in 4 is now all
> lost
> *Why?*
> for case 1
> Since preserve splits in truncate table procedure will not change the
> regioninfo, when log replay happens, the 'unflushed' data will restore back
> to the region
> for case 2
> since the flushedSequenceIdByRegion are stored in Master in a map with the
> region's encodedName. Although the table is truncated, the region's name is
> not changed since we chose to preserve the splits. So after truncate the
> table, the region's sequenceid is reset in the regionserver, but not reset in
> master. When flush comes and report to master, master will reject the update
> of sequenceid since the new one is smaller than the old one. The same happens
> in log replay, all the edits writen in 4 will be skipped since they have a
> smaller seqid
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505283#comment-15505283
]
Ted Yu commented on HBASE-16646:
Failed tests passed locally with patch:
{code}
Running
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 149.521 sec -
in org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.411 sec - in
org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.199 sec - in
org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.replication.TestMasterReplication
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 264.055 sec -
in org.apache.hadoop.hbase.replication.TestMasterReplication
{code}
> Enhance LoadIncrementalHFiles API to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt, 16646.v3.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with
> loading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16657: -- Priority: Minor (was: Major) > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Priority: Minor > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
Gary Helmling created HBASE-16657: - Summary: Expose per-region last major compaction timestamp in RegionServer UI Key: HBASE-16657 URL: https://issues.apache.org/jira/browse/HBASE-16657 Project: HBase Issue Type: Improvement Components: regionserver, UI Reporter: Gary Helmling HBASE-12859 added some tracking for the last major compaction completed for each region. However, this is currently only exposed through the cluster status reporting and the Admin API. Since the regionserver is already reporting this information, it would be nice to fold it in somewhere to the region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11236) Last flushed sequence id is ignored by ServerManager
[
https://issues.apache.org/jira/browse/HBASE-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505258#comment-15505258
]
Allan Yang commented on HBASE-11236:
ignore flushed sequence id can cause data loss, please ref to
https://issues.apache.org/jira/browse/HBASE-16649?focusedCommentId=15502490=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15502490
> Last flushed sequence id is ignored by ServerManager
>
>
> Key: HBASE-11236
> URL: https://issues.apache.org/jira/browse/HBASE-11236
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>
> I got lots of error messages like this:
> {quote}
> 2014-05-22 08:58:59,793 DEBUG [RpcServer.handler=1,port=20020]
> master.ServerManager: RegionServer
> a2428.halxg.cloudera.com,20020,1400742071109 indicates a last flushed
> sequence id (numberOfStores=9, numberOfStorefiles=2,
> storefileUncompressedSizeMB=517, storefileSizeMB=517,
> compressionRatio=1., memstoreSizeMB=0, storefileIndexSizeMB=0,
> readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=34,
> totalStaticIndexSizeKB=381, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
> currentCompactedKVs=0, compactionProgressPct=NaN) that is less than the
> previous last flushed sequence id (605446) for region
> IntegrationTestBigLinkedList,
> �A��*t�^FU�2��0,1400740489477.a44d3e309b5a7e29355f6faa0d3a4095. Ignoring.
> {quote}
> RegionLoad.toString doesn't print out the last flushed sequence id passed in.
> Why is it less than the previous one?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-16257:
-
Attachment: HBASE-16257-v5.patch
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch, HBASE-16257-v5.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505232#comment-15505232
]
Jerry He commented on HBASE-16257:
--
bq. If it is not the case, then we can commit this patch as it is.
Is this +1 [~enis]?
Thanks for the comment, [~ashish singhi]
I tested with a local standalone instance and on a Keberos cluster. All looks
good.
The directories are created or modified with the correct permissions:
{noformat}
hadoop fs -ls /apps/hbase/
drwx--x--x - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data
hadoop fs -ls /apps/hbase/data
drwx-- - hbase hdfs 0 2016-09-19 11:25 /apps/hbase/data/.hbck
drwx-- - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data/.tmp
drwx-- - hbase hdfs 0 2016-09-19 18:01
/apps/hbase/data/MasterProcWALs
drwx-- - hbase hdfs 0 2016-09-19 13:48 /apps/hbase/data/WALs
drwx-- - hbase hdfs 0 2016-09-19 14:01 /apps/hbase/data/archive
drwx-- - hbase hdfs 0 2016-06-17 15:25 /apps/hbase/data/corrupt
drwx-- - hbase hdfs 0 2016-06-09 12:36 /apps/hbase/data/data
-rw--- 3 hbase hdfs 42 2016-06-09 12:35 /apps/hbase/data/hbase.id
-rw--- 3 hbase hdfs 7 2016-06-09 12:35
/apps/hbase/data/hbase.version
drwx-- - hbase hdfs 0 2016-09-19 11:25 /apps/hbase/data/mobdir
drwx-- - hbase hdfs 0 2016-09-19 17:59 /apps/hbase/data/oldWALs
drwx--x--x - hbase hdfs 0 2016-09-19 17:34 /apps/hbase/data/staging
{noformat}
Ran a few bulk loads as well.
v5 adds back creating staging dir in SecureBulkLoadManager in the region server
side if it does not exist, so that it is ok to upgrade region server before
master.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505215#comment-15505215
]
Hadoop QA commented on HBASE-16646:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 160m 41s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 4s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
45m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 121m 46s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 354m 13s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| | hadoop.hbase.replication.TestMasterReplication |
| | hadoop.hbase.master.TestAssignmentManagerMetrics |
| | hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin1 |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClient |
| |
org.apache.hadoop.hbase.client.TestCloneSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829249/16646.v3.txt |
| JIRA Issue | HBASE-16646 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d38584043c98 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2eac0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
[jira] [Updated] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16655:
---
Attachment: 16655.addendum
The addendum allows progress command to retrieve on-going backup session.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.addendum, 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution
[ https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16587: Attachment: HBASE-16587-v3.patch > Procedure v2 - Cleanup suspended proc execution > --- > > Key: HBASE-16587 > URL: https://issues.apache.org/jira/browse/HBASE-16587 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16587-v0.patch, HBASE-16587-v1.patch, > HBASE-16587-v2.patch, HBASE-16587-v3.patch > > > for procedures like the assignment or the lock one we need to be able to hold > on locks while suspended. At the moment the way to do that is up to the proc > implementation. This patch moves the logic to the base Procedure and > ProcedureExecutor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[
https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505189#comment-15505189
]
Hadoop QA commented on HBASE-16649:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 5s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 7s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestCatalogJanitor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829117/HBASE-16649-v0.patch |
| JIRA Issue | HBASE-16649 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 51e5ae5b0817 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b2eac0d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3610/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3610/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3610/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3610/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Truncate table with splits preserved can cause both data loss and truncated
> data appeared again
>
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[
https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505165#comment-15505165
]
Hadoop QA commented on HBASE-16264:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 160 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 29m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 44s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 5s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
25s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 33s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 33s {color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
10s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 2299 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 7s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 29s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 6s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 58s
{color} | {color:green} root generated 0 new + 19 unchanged - 1 fixed = 19
total (was 20) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} hbase-client generated 0 new + 13 unchanged - 1 fixed =
13 total (was 14) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} hbase-examples in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 11s
{color} | {color:red} hbase-procedure generated 6 new + 0 unchanged - 0 fixed =
6 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 6s
{color}
[jira] [Commented] (HBASE-16554) Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted.
[ https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505128#comment-15505128 ] Hudson commented on HBASE-16554: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1636 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1636/]) HBASE-16554 Rebuild WAL tracker if trailer is corrupted. (appy: rev b2eac0da33c4161aa8188213171afb03b72048a4) * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFile.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java > Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted. > - > > Key: HBASE-16554 > URL: https://issues.apache.org/jira/browse/HBASE-16554 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16554.master.001.patch, > HBASE-16554.master.002.patch, tracker-rebuild.patch > > > If the last wal was closed cleanly, the global tracker will be the last wal > tracker (no rebuild needed) > if the last wal does not have a tracker (corrupted/master-killed). on load() > we will rebuild the global tracker. > To compute quickly which files should be deleted, we also want the tracker of > each file. > if the wal was closed properly and has a tracker we are good, if not we need > to rebuild the tracker for that file. > each file tracker contains a bitmap about what is in the wal (the updated > bitmap), which is easy to compute just by reading each entry of the wal. > The 'deleted' bitmap keeps track of the "running procedures" up to that wal, > however, it's not required by WAL cleaner, so we don't bother recovering it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505115#comment-15505115 ] deepankar edited comment on HBASE-16630 at 9/20/16 12:17 AM: - Yup looks like this is exactly what we want, the idea is to pick a random slab and evict all elements from it, but we can be a little bit more clever here and pick the slabs that would have least number of elements with in them so that we will do least number of reads to re-cache the elements from them. This idea is simple and could be done easily with current code structure also I think with small amount of code. [~vrodionov] what do you think about this heuristic ? I think this will also address the concerns [~zjushch] raised [~vrodionov]Thanks for the idea was (Author: dvdreddy): Yup looks like this is exactly what we want, the idea is to pick a random slab and evict all elements from it, but we can be a little bit more clever here and pick the slabs that would have least number of elements with in them so that we will do least number of reads to re-cache the elements from them. This idea is simple and could be done easily with current code structure also I think with small amount of code. [~vrodionov] do you think ? This will also address the concerns [~zjushch] raised [~vrodionov]Thanks for the idea > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505118#comment-15505118
]
Ted Yu commented on HBASE-16655:
{code}
public static final String PROGRESS_CMD_USAGE = "Usage: hbase backup progress
\n"
+ " backupIdbackup image id\n";
{code}
When backup session is in progress, user can obtain backupId from Proc V2 tab
of master UI.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505115#comment-15505115 ] deepankar commented on HBASE-16630: --- Yup looks like this is exactly what we want, the idea is to pick a random slab and evict all elements from it, but we can be a little bit more clever here and pick the slabs that would have least number of elements with in them so that we will do least number of reads to re-cache the elements from them. This idea is simple and could be done easily with current code structure also I think with small amount of code. [~vrodionov] do you think ? This will also address the concerns [~zjushch] raised [~vrodionov]Thanks for the idea > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505096#comment-15505096
]
Vladimir Rodionov edited comment on HBASE-16655 at 9/20/16 12:07 AM:
-
[~tedyu], the default behavior of progress command w/o attributes is to lookup
current RUNNING backup session from hbase:backup table and present progress
info for that session. We need to preserve it, as since, when session is in
progress, there is no easy way to get backupID of this session (we return it on
a backup completion).
was (Author: vrodionov):
[~tedyu], the default behavior of progress command w/o attributes is to lookup
current RUNNING backup session from hbase:backup table and present progress
info for that session. Is this still valid?
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505096#comment-15505096
]
Vladimir Rodionov commented on HBASE-16655:
---
[~tedyu], the default behavior of progress command w/o attributes is to lookup
current RUNNING backup session from hbase:backup table and present progress
info for that session. Is this still valid?
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505090#comment-15505090 ] Vladimir Rodionov commented on HBASE-16630: --- Google "slab calcification problem". This is exactly BucketCache "fragmentation issue". > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16264: -- Attachment: HBASE-16264.master.011.patch > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch, HBASE-16264.master.011.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[ https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16649: Assignee: Matteo Bertozzi Status: Patch Available (was: Open) > Truncate table with splits preserved can cause both data loss and truncated > data appeared again > --- > > Key: HBASE-16649 > URL: https://issues.apache.org/jira/browse/HBASE-16649 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.3 >Reporter: Allan Yang >Assignee: Matteo Bertozzi > Attachments: HBASE-16649-v0.patch > > > Since truncate table with splits preserved will delete hfiles and use the > previous regioninfo. It can cause odd behaviors > - Case 1: *Data appeared after truncate* > reproduce procedure： > 1. create a table, let's say 'test' > 2. write data to 'test', make sure memstore of 'test' is not empty > 3. truncate 'test' with splits preserved > 4. kill the regionserver hosting the region(s) of 'test' > 5. start the regionserver, now it is the time to witness the miracle! the > truncated data appeared in table 'test' > - Case 2: *Data loss* > reproduce procedure: > 1. create a table, let's say 'test' > 2. write some data to 'test', no matter how many > 3. truncate 'test' with splits preserved > 4. restart the regionserver to reset the seqid > 5. write some data, but less than 2 since we don't want the seqid to run over > the one in 2 > 6. kill the regionserver hosting the region(s) of 'test' > 7. restart the regionserver. Congratulations! the data writen in 4 is now all > lost > *Why?* > for case 1 > Since preserve splits in truncate table procedure will not change the > regioninfo, when log replay happens, the 'unflushed' data will restore back > to the region > for case 2 > since the flushedSequenceIdByRegion are stored in Master in a map with the > region's encodedName. Although the table is truncated, the region's name is > not changed since we chose to preserve the splits. So after truncate the > table, the region's sequenceid is reset in the regionserver, but not reset in > master. When flush comes and report to master, master will reject the update > of sequenceid since the new one is smaller than the old one. The same happens > in log replay, all the edits writen in 4 will be skipped since they have a > smaller seqid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name (by default) or can be set by user
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Description: Default backup set name is "backup". If we do backup for a backup set "SomeSetName", by default, backup id will be generated in a form: *SomeSetName_timestamp*. User can override default behavior by providing backup prefix name in a command - line: *-backup_prefix * The goal is to separate backup images between different sets and backup destinations. The history command will be updated and the new command - merge will use this backup names (prefixes) was: Default backup set name is "backup". If we do backup for a backup set "SomeSetName", by default, backup id will be generated in a form: *SomeSetName_timestamp*. User can override default behavior by providing backup prefix name in a command - line: *-backup_prefix * > BackupID must include backup set name (by default) or can be set by user > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > User can override default behavior by providing backup prefix name in a > command - line: > *-backup_prefix * > The goal is to separate backup images between different sets and backup > destinations. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name (by default) or can be set by user
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Summary: BackupID must include backup set name (by default) or can be set by user (was: BackupID must include backup set name (by default) or can be by user ) > BackupID must include backup set name (by default) or can be set by user > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > User can override default behavior by providing backup prefix name in a > command - line: > *-backup_prefix * -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16656) BackupID must include backup set name (by default) or can be by user
Vladimir Rodionov created HBASE-16656: - Summary: BackupID must include backup set name (by default) or can be by user Key: HBASE-16656 URL: https://issues.apache.org/jira/browse/HBASE-16656 Project: HBase Issue Type: Improvement Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Default backup set name is "backup". If we do backup for a backup set "SomeSetName", by default, backup id will be generated in a form: *SomeSetName_timestamp*. User can override default behavior by providing backup prefix name in a command - line: *-backup_prefix * -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution
[
https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504852#comment-15504852
]
Hadoop QA commented on HBASE-16587:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.master.TestTableLockManager |
| | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
| | org.apache.hadoop.hbase.master.TestWarmupRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829258/HBASE-16587-v2.patch |
| JIRA Issue | HBASE-16587 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile xml |
| uname | Linux fc1e069804ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair
[
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504836#comment-15504836
]
Ted Yu commented on HBASE-16647:
lgtm
> hbck should do offline reference repair before online repair
>
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Attachments: HBASE-16647-v2.patch, HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files
> failed regions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-16655.
Resolution: Fixed
Hadoop Flags: Reviewed
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504819#comment-15504819
]
Ted Yu commented on HBASE-16655:
Both TestBackupCommandLineTool and TestBackupDescribe passed based on patch v3.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504791#comment-15504791
]
Vladimir Rodionov commented on HBASE-16655:
---
Yes, I know. That was intended behavior : if no id specified for progress - we
will retrieve this backup id from system table (RUNNING state). But I think
this should be changed to default behavior - if no id, then print usage. Go
ahead and combine patches, [~tedyu].
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16655:
---
Attachment: 16655.v3.txt
Patch v3 combines addendum 6 from HBASE-16620 .
Vlad:
I will give you credit when committing the patch.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt, 16655.v3.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504765#comment-15504765
]
Ted Yu commented on HBASE-16655:
Addendum 6 would result in retrieving progress for null backup Id if there is
no backup Id specified on the command line:
{code}
"main" #1 prio=5 os_prio=31 tid=0x7fe493006800 nid=0x1703 waiting on
condition [0x70217000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at java.lang.Thread.sleep(Thread.java:340)
at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
at
org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:158)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:373)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:620)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:479)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:165)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:577)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:556)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:58)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateMeta(ConnectionImplementation.java:765)
- locked <0x00078a5e7d48> (a java.lang.Object)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:732)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:185)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:829)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:735)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:298)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:185)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:311)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:286)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:162)
at
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at
org.apache.hadoop.hbase.client.ClientSimpleScanner.(ClientSimpleScanner.java:40)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:381)
at
org.apache.hadoop.hbase.backup.impl.BackupSystemTable.getBackupContexts(BackupSystemTable.java:355)
at
org.apache.hadoop.hbase.client.HBaseBackupAdmin.getProgress(HBaseBackupAdmin.java:88)
at
org.apache.hadoop.hbase.backup.impl.BackupCommands$ProgressCommand.execute(BackupCommands.java:367)
at
org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
at
org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
at
org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.backup.TestBackupCommandLineTool.testBackupDriverProgressHelp(TestBackupCommandLineTool.java:176)
{code}
Let me make a patch that combines addendum 6 and my patch.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project:
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504754#comment-15504754
]
Devaraj Das commented on HBASE-16604:
-
LGTM
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 2016-09-09 16:27:15,635 DEBUG
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833] ipc.CallRunner(110):
> B.fifo.QRpcServer.handler=5,queue=0,port=51833: callId: 26 service:
> ClientService methodName: Scan size: 26 connection: 192.168.42.75:51903
> java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for
> reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:224)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:312)
> at
>
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504749#comment-15504749 ] stack commented on HBASE-16264: --- Patch is just 2MB when I strip the generated content. Updated rb: https://reviews.apache.org/r/51345/ > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504716#comment-15504716 ] stack commented on HBASE-16264: --- Good idea for getting patch up on rb. Thanks. > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504713#comment-15504713
]
Vladimir Rodionov commented on HBASE-16655:
---
[~tedyu] please keep only "hbase backup describe with incorrect backup id
results in NPE" related fix and new test in the patch. I have taken care about
the rest (progress / describe with no backupId) in HBASE-16620 addendum6 patch.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504688#comment-15504688 ] Vladimir Rodionov edited comment on HBASE-16620 at 9/19/16 9:03 PM: addendum6 adds more unit tests. cc:[~tedyu] was (Author: vrodionov): addendum6 adds more unit tests. > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > 16620.addendum4, 16620.addendum5, HBASE-16620-addendum6.patch, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16620) Fix backup command-line tool usability issues
[ https://issues.apache.org/jira/browse/HBASE-16620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16620: -- Attachment: HBASE-16620-addendum6.patch addendum6 adds more unit tests. > Fix backup command-line tool usability issues > - > > Key: HBASE-16620 > URL: https://issues.apache.org/jira/browse/HBASE-16620 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Attachments: 16620-v2.patch, 16620-v4.patch, 16620.addendum, > 16620.addendum4, 16620.addendum5, HBASE-16620-addendum6.patch, > HBASE-16620-v1.patch, HBASE-16620-v3.patch, HBASE-16620.add.patch > > > We need to address issues found by [~saint@gmail.com] > https://issues.apache.org/jira/browse/HBASE-7912?focusedCommentId=15484865=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15484865 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-7912) HBase Backup/Restore Based on HBase Snapshot
[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Welsch updated HBASE-7912: Attachment: Backup-and-Restore-Apache_19Sep2016.pdf I am attaching revised doc of the feature. The revisions are for the following: --correct the backupId argument spelling and format on the CLI --to remove the hbase backup cancel functionality from the doc because it's not yet supported --list limitations of the current HBase backup-and-restore functionality > HBase Backup/Restore Based on HBase Snapshot > > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task >Reporter: Richard Ding >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > Backup-and-Restore-Apache_9Sep2016.pdf, HBaseBackupAndRestore - v0.8.pdf, > HBaseBackupAndRestore -0.91.pdf, HBaseBackupAndRestore-v0.9.pdf, > HBaseBackupAndRestore.pdf, HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf, > HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf, > HBaseBackupRestore-Jira-7912-v6.pdf, HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full backup. > We would support the following key features and capabilities. > * Full backup uses HBase snapshot to capture HFiles. > * Use HBase WALs to capture incremental changes, but we use bulk load of > HFiles for fast incremental restore. > * Support single table or a set of tables, and column family level backup and > restore. > * Restore to different table names. > * Support adding additional tables or CF to backup set without
[jira] [Updated] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15448: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch, Vlad. > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch, HBASE-15448-v5.patch, HBASE-15448-v6.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504611#comment-15504611
]
Enis Soztutar commented on HBASE-16604:
---
[~devaraj] any more concerns with the patch. The test failures are due to
flakiness.
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 2016-09-09 16:27:15,635 DEBUG
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833] ipc.CallRunner(110):
> B.fifo.QRpcServer.handler=5,queue=0,port=51833: callId: 26 service:
> ClientService methodName: Scan size: 26 connection: 192.168.42.75:51903
> java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for
> reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:224)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
> at
>
[jira] [Commented] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[
https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504591#comment-15504591
]
Ted Yu commented on HBASE-15448:
I tried reducing the number of additional rows and TestIncrementalBackup passes
with:
{code}
int ADD_ROWS = 99;
{code}
> HBase Backup Phase 3: Restore optimization 2
>
>
> Key: HBASE-15448
> URL: https://issues.apache.org/jira/browse/HBASE-15448
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch,
> HBASE-15448-v4.patch, HBASE-15448-v5.patch, HBASE-15448-v6.patch
>
>
> JIRA opened to continue work on restore optimization.
> This will focus on the following
> # During incremental backup image restore - restoring full backup into region
> boundaries of the most recent incremental backup image.
> # Combining multiple tables into single M/R job
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16264: -- Attachment: HBASE-16264.master.010.patch > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch, > HBASE-16264.master.010.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504536#comment-15504536 ] deepankar commented on HBASE-16630: --- This is very good idea to make sure all the buckets have freeCount, since there are no free counts, one way to enforce this would be force the eviction from that BucketSizeInfo which would lead to eviction of hot blocks also if the current buckets of that BucketSize is small and also it would also lockup the unused and fragmented space from the other bucketSizes until a compaction or something else frees them up. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504524#comment-15504524 ] Matteo Bertozzi commented on HBASE-16264: - remove the generated *Protos.java from the patch you are uploading to rb. we are not going to review those anyway. and they should be most of the patch size > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504518#comment-15504518 ] deepankar commented on HBASE-16630: --- Ha, nice point, I missed this, but this would still not guarantee, that we would freeup the buckets needed for the given BucketSize in a single run, it might need several runs free-up a completely free bucket and that would transfer into the our hot BucketSize. One other reason it might be slow is that bytesToFreeWithExtra will depend on the size of the currently hot BucketSize (for which there are no free blocks) and that is usually small and the transition might be slow. Another problem is that due to sparsely occupied buckets we might encounter keep ending up in the freeSpace method again and again. > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16264: -- Release Note: Shade/relocate the protobuf hbase uses internally. All core now refers to new module added in this patch, hbase-protocol-shaded. Coprocessor Endpoints carry-on with references to the original hbase-protocol module. (was: Shade/relocate the protobuf hbase uses internally. Even though our API references protobufs -- to support Coprocessor Endpoints -- Coprocessor Endpoints should still work (it is a bug if they do not).) > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504507#comment-15504507 ] stack commented on HBASE-16264: --- Back up to 14.5MB. Won't go up on RB. Lets see how this run does. Will clean up the patch then probably put it in a branch. It is a simple patch in the end. Mostly just resetting imports and adding a new module that has a bunch of overlap with hbase-protocol. I updated https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, HBASE-16264.master.001.patch, > HBASE-16264.master.002.patch, HBASE-16264.master.003.patch, > HBASE-16264.master.004.patch, HBASE-16264.master.005.patch, > HBASE-16264.master.006.patch, HBASE-16264.master.007.patch, > HBASE-16264.master.008.patch, HBASE-16264.master.009.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[
https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504473#comment-15504473
]
Hadoop QA commented on HBASE-15448:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-15448 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829259/HBASE-15448-v6.patch |
| JIRA Issue | HBASE-15448 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3607/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> HBase Backup Phase 3: Restore optimization 2
>
>
> Key: HBASE-15448
> URL: https://issues.apache.org/jira/browse/HBASE-15448
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Critical
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch,
> HBASE-15448-v4.patch, HBASE-15448-v5.patch, HBASE-15448-v6.patch
>
>
> JIRA opened to continue work on restore optimization.
> This will focus on the following
> # During incremental backup image restore - restoring full backup into region
> boundaries of the most recent incremental backup image.
> # Combining multiple tables into single M/R job
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15448) HBase Backup Phase 3: Restore optimization 2
[ https://issues.apache.org/jira/browse/HBASE-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15448: -- Attachment: HBASE-15448-v6.patch v6. > HBase Backup Phase 3: Restore optimization 2 > > > Key: HBASE-15448 > URL: https://issues.apache.org/jira/browse/HBASE-15448 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15448-v1.patch, HBASE-15448-v2.patch, > HBASE-15448-v4.patch, HBASE-15448-v5.patch, HBASE-15448-v6.patch > > > JIRA opened to continue work on restore optimization. > This will focus on the following > # During incremental backup image restore - restoring full backup into region > boundaries of the most recent incremental backup image. > # Combining multiple tables into single M/R job -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504455#comment-15504455 ] Andrew Purtell commented on HBASE-15984: That is my take too. So I'd say commit back to 1.2, let [~ndimiduk] decide if he wants it in 1.1 and I'll pick back to 0.98. Sound like a plan? > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504436#comment-15504436 ] Sean Busbey commented on HBASE-15984: - Since we're talking about data loss, I originally wanted to push this for all branches. I'm fine leaving out the metrics additions on maintenance release lines if folks prefer, but I've always interpreted the operational section of the compatibility promises as saying we wouldn't remove metrics in a maintenance release rather than that we wouldn't add them. > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16554) Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted.
[ https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16554: - Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted. > - > > Key: HBASE-16554 > URL: https://issues.apache.org/jira/browse/HBASE-16554 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16554.master.001.patch, > HBASE-16554.master.002.patch, tracker-rebuild.patch > > > If the last wal was closed cleanly, the global tracker will be the last wal > tracker (no rebuild needed) > if the last wal does not have a tracker (corrupted/master-killed). on load() > we will rebuild the global tracker. > To compute quickly which files should be deleted, we also want the tracker of > each file. > if the wal was closed properly and has a tracker we are good, if not we need > to rebuild the tracker for that file. > each file tracker contains a bitmap about what is in the wal (the updated > bitmap), which is easy to compute just by reading each entry of the wal. > The 'deleted' bitmap keeps track of the "running procedures" up to that wal, > however, it's not required by WAL cleaner, so we don't bother recovering it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16554) Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted.
[ https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16554: - Description: If the last wal was closed cleanly, the global tracker will be the last wal tracker (no rebuild needed) if the last wal does not have a tracker (corrupted/master-killed). on load() we will rebuild the global tracker. To compute quickly which files should be deleted, we also want the tracker of each file. if the wal was closed properly and has a tracker we are good, if not we need to rebuild the tracker for that file. each file tracker contains a bitmap about what is in the wal (the updated bitmap), which is easy to compute just by reading each entry of the wal. was: If the last wal was closed cleanly, the global tracker will be the last wal tracker (no rebuild needed) if the last wal does not have a tracker (corrupted/master-killed). on load() we will rebuild the global tracker. To compute quickly which files should be deleted, we also want the tracker of each file. if the wal was closed properly and has a tracker we are good, if not we need to rebuild the tracker for that file. each file tracker contains a bitmap about what is in the wal (the updated bitmap), which is easy to compute just by reading each entry of the wal. and a bitmap that keeps track of the "running procedures" up to that wal (the deleted bitmap). The delete bitmap requires a bit of post read-all-wals work. and it will basically require to AND the deleted bitmap of wal\(i\) and wal(i-1) > Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted. > - > > Key: HBASE-16554 > URL: https://issues.apache.org/jira/browse/HBASE-16554 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16554.master.001.patch, > HBASE-16554.master.002.patch, tracker-rebuild.patch > > > If the last wal was closed cleanly, the global tracker will be the last wal > tracker (no rebuild needed) > if the last wal does not have a tracker (corrupted/master-killed). on load() > we will rebuild the global tracker. > To compute quickly which files should be deleted, we also want the tracker of > each file. > if the wal was closed properly and has a tracker we are good, if not we need > to rebuild the tracker for that file. > each file tracker contains a bitmap about what is in the wal (the updated > bitmap), which is easy to compute just by reading each entry of the wal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16554) Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted.
[ https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16554: - Summary: Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted. (was: Procedure V2 - handle corruption of WAL trailer) > Procedure V2 - Recover 'updated' part of WAL tracker if trailer is corrupted. > - > > Key: HBASE-16554 > URL: https://issues.apache.org/jira/browse/HBASE-16554 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16554.master.001.patch, > HBASE-16554.master.002.patch, tracker-rebuild.patch > > > If the last wal was closed cleanly, the global tracker will be the last wal > tracker (no rebuild needed) > if the last wal does not have a tracker (corrupted/master-killed). on load() > we will rebuild the global tracker. > To compute quickly which files should be deleted, we also want the tracker of > each file. > if the wal was closed properly and has a tracker we are good, if not we need > to rebuild the tracker for that file. > each file tracker contains a bitmap about what is in the wal (the updated > bitmap), which is easy to compute just by reading each entry of the wal. and > a bitmap that keeps track of the "running procedures" up to that wal (the > deleted bitmap). The delete bitmap requires a bit of post read-all-wals work. > and it will basically require to AND the deleted bitmap of wal\(i\) and > wal(i-1) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504404#comment-15504404 ] Ted Yu commented on HBASE-14123: [~stack]: Looking forward to your feedback / code review. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v3.txt, 14123-master.v5.txt, > 14123-master.v6.txt, 14123-master.v7.txt, 14123-master.v8.txt, > 14123-master.v9.txt, 14123-v14.txt, HBASE-14123-for-7912-v1.patch, > HBASE-14123-for-7912-v6.patch, HBASE-14123-v1.patch, HBASE-14123-v10.patch, > HBASE-14123-v11.patch, HBASE-14123-v12.patch, HBASE-14123-v13.patch, > HBASE-14123-v15.patch, HBASE-14123-v16.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch, HBASE-14123-v4.patch, HBASE-14123-v5.patch, > HBASE-14123-v6.patch, HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16554) Procedure V2 - handle corruption of WAL trailer
[ https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16554: - Fix Version/s: 2.0.0 > Procedure V2 - handle corruption of WAL trailer > --- > > Key: HBASE-16554 > URL: https://issues.apache.org/jira/browse/HBASE-16554 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-16554.master.001.patch, > HBASE-16554.master.002.patch, tracker-rebuild.patch > > > If the last wal was closed cleanly, the global tracker will be the last wal > tracker (no rebuild needed) > if the last wal does not have a tracker (corrupted/master-killed). on load() > we will rebuild the global tracker. > To compute quickly which files should be deleted, we also want the tracker of > each file. > if the wal was closed properly and has a tracker we are good, if not we need > to rebuild the tracker for that file. > each file tracker contains a bitmap about what is in the wal (the updated > bitmap), which is easy to compute just by reading each entry of the wal. and > a bitmap that keeps track of the "running procedures" up to that wal (the > deleted bitmap). The delete bitmap requires a bit of post read-all-wals work. > and it will basically require to AND the deleted bitmap of wal\(i\) and > wal(i-1) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16554) Procedure V2 - handle corruption of WAL trailer
[
https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504392#comment-15504392
]
Hadoop QA commented on HBASE-16554:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 10s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829246/HBASE-16554.master.002.patch
|
| JIRA Issue | HBASE-16554 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a454d0861fe7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c5b8aab |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3605/testReport/ |
| modules | C: hbase-procedure U: hbase-procedure |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3605/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Procedure V2 - handle corruption of WAL trailer
> ---
>
> Key: HBASE-16554
> URL: https://issues.apache.org/jira/browse/HBASE-16554
> Project: HBase
> Issue Type: Sub-task
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16554.master.001.patch,
> HBASE-16554.master.002.patch, tracker-rebuild.patch
>
>
> If the last wal was closed cleanly, the global tracker will
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504374#comment-15504374 ] Andrew Purtell commented on HBASE-15984: +1 Patch also introduces metrics changes. That's maybe 50% of the bulk of the diff. Are we ok with metrics changes in patch releases? I think it's analogous to an API addition. After an upgrade, a change to a metrics system to report on the new metrics will fail to receive data after a downgrade. In my opinion this is fine, certainly for a minor release, so commit to master, branch-1, and 0.98 will be fine. (That 0.98 commit will need work to add Hadoop 1 compat metrics, I can do that.) For 1.1 and 1.2 I defer to the RMs [~ndimiduk] and [~busbey] > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input
[ https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16646: --- Attachment: 16646.v3.txt Patch v3 adds test which exercises the new API. > Enhance LoadIncrementalHFiles API to accept store file paths as input > - > > Key: HBASE-16646 > URL: https://issues.apache.org/jira/browse/HBASE-16646 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt, 16646.v3.txt > > > Currently LoadIncrementalHFiles takes the directory (output path) as input > parameter. > In some scenarios (incremental restore of bulk loaded hfiles), the List of > paths to hfiles is known. > LoadIncrementalHFiles can take the List as input parameter and proceed with > loading. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504314#comment-15504314 ] Sean Busbey commented on HBASE-15984: - bump? this provides an incremental improvement and stops us from silently throwing away data. > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16554) Procedure V2 - handle corruption of WAL trailer
[ https://issues.apache.org/jira/browse/HBASE-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16554: - Attachment: HBASE-16554.master.002.patch > Procedure V2 - handle corruption of WAL trailer > --- > > Key: HBASE-16554 > URL: https://issues.apache.org/jira/browse/HBASE-16554 > Project: HBase > Issue Type: Sub-task >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16554.master.001.patch, > HBASE-16554.master.002.patch, tracker-rebuild.patch > > > If the last wal was closed cleanly, the global tracker will be the last wal > tracker (no rebuild needed) > if the last wal does not have a tracker (corrupted/master-killed). on load() > we will rebuild the global tracker. > To compute quickly which files should be deleted, we also want the tracker of > each file. > if the wal was closed properly and has a tracker we are good, if not we need > to rebuild the tracker for that file. > each file tracker contains a bitmap about what is in the wal (the updated > bitmap), which is easy to compute just by reading each entry of the wal. and > a bitmap that keeps track of the "running procedures" up to that wal (the > deleted bitmap). The delete bitmap requires a bit of post read-all-wals work. > and it will basically require to AND the deleted bitmap of wal\(i\) and > wal(i-1) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504211#comment-15504211
]
Enis Soztutar commented on HBASE-16257:
---
If source is 2.0 with this patch, but sink is 1.x without this patch, then the
sink will not create the staging dir under root. However, on the source side,
we will still be accessing the staging dir under root on the sink fs.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16447) Replication by namespaces config in peer
[ https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504198#comment-15504198 ] Enis Soztutar commented on HBASE-16447: --- Thanks. Left a comment there. > Replication by namespaces config in peer > > > Key: HBASE-16447 > URL: https://issues.apache.org/jira/browse/HBASE-16447 > Project: HBase > Issue Type: New Feature > Components: Replication >Affects Versions: 2.0.0 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, > HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch, > HBASE-16447-v5.patch > > > Now we only config table cfs in peer. But in our production cluster, there > are a dozen of namespace and every namespace has dozens of tables. It was > complicated to config all table cfs in peer. For some namespace, it need > replication all tables to other slave cluster. It will be easy to config if > we support replication by namespace. Suggestions and discussions are welcomed. > Review board: https://reviews.apache.org/r/51521/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11386) Replication#table,CF config will be wrong if the table name includes namespace
[
https://issues.apache.org/jira/browse/HBASE-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504197#comment-15504197
]
Enis Soztutar commented on HBASE-11386:
---
We should resolve this one as won't fix in favor of a backport of HBASE-11393
to branch-1.
> Replication#table,CF config will be wrong if the table name includes namespace
> --
>
> Key: HBASE-11386
> URL: https://issues.apache.org/jira/browse/HBASE-11386
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Qianxi Zhang
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE_11386_trunk_v1.patch, HBASE_11386_trunk_v2.patch
>
>
> Now we can config the table and CF in Replication, but I think the parse will
> be wrong if the table name includes namespace
> ReplicationPeer#parseTableCFsFromConfig(line 125)
> {code}
> Map tableCFsMap = null;
> // parse out (table, cf-list) pairs from tableCFsConfig
> // format: "table1:cf1,cf2;table2:cfA,cfB"
> String[] tables = tableCFsConfig.split(";");
> for (String tab : tables) {
> // 1 ignore empty table config
> tab = tab.trim();
> if (tab.length() == 0) {
> continue;
> }
> // 2 split to "table" and "cf1,cf2"
> // for each table: "table:cf1,cf2" or "table"
> String[] pair = tab.split(":");
> String tabName = pair[0].trim();
> if (pair.length > 2 || tabName.length() == 0) {
> LOG.error("ignore invalid tableCFs setting: " + tab);
> continue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504158#comment-15504158
]
Ashish Singhi commented on HBASE-16257:
---
It need not. It's HFileReplicator which will create and access the directory in
the sink cluster and it will run in sink cluster so there should not be any
problem.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16655:
---
Attachment: 16655.v2.txt
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16655:
---
Attachment: (was: 16655.v2.txt)
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504151#comment-15504151
]
Umesh Agashe commented on HBASE-16294:
--
Looking into it.
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe reassigned HBASE-16294:
Assignee: Umesh Agashe
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504140#comment-15504140
]
Enis Soztutar commented on HBASE-16257:
---
In secure 1.x clusters, root dir is protected with 700. So unless the user is
the same, source cluster cannot create or access a directory under the sink
cluster.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16257:
--
Comment: was deleted
(was: In secure 1.x clusters, root dir is protected with 700. So unless the
user is the same, source cluster cannot create or access a directory under the
sink cluster. )
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504142#comment-15504142
]
Enis Soztutar commented on HBASE-16257:
---
In secure 1.x clusters, root dir is protected with 700. So unless the user is
the same, source cluster cannot create or access a directory under the sink
cluster.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch,
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504131#comment-15504131
]
Ted Yu commented on HBASE-16655:
Modified test in TestBackupDescribe for the case of incorrect backup Id.
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
[
https://issues.apache.org/jira/browse/HBASE-16655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16655:
---
Attachment: 16655.v2.txt
> hbase backup describe with incorrect backup id results in NPE
> -
>
> Key: HBASE-16655
> URL: https://issues.apache.org/jira/browse/HBASE-16655
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: backup
> Attachments: 16655.v2.txt
>
>
> If the user supplies backup Id which is non-existent, we would get
> NullPointerException :
> {code}
> 2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
> command-line tool
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
> at
> org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
> at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
> at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16655) hbase backup describe with incorrect backup id results in NPE
Ted Yu created HBASE-16655:
--
Summary: hbase backup describe with incorrect backup id results in
NPE
Key: HBASE-16655
URL: https://issues.apache.org/jira/browse/HBASE-16655
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
If the user supplies backup Id which is non-existent, we would get
NullPointerException :
{code}
2016-09-19 10:26:50,771 ERROR [main] backup.BackupDriver(173): Error running
command-line tool
java.lang.NullPointerException
at
org.apache.hadoop.hbase.backup.impl.BackupCommands$DescribeCommand.execute(BackupCommands.java:329)
at
org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:114)
at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:135)
at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:171)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.backup.TestBackupDescribe.testBackupDescribe(TestBackupDescribe.java:67)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503978#comment-15503978
]
Hadoop QA commented on HBASE-14882:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 36s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 38s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 52s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 29s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829228/HBASE-14882.master.002.patch
|
| JIRA Issue | HBASE-14882 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 4ac2471e3214 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c5b8aab |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3604/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3604/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3604/console |
|
[jira] [Commented] (HBASE-16649) Truncate table with splits preserved can cause both data loss and truncated data appeared again
[ https://issues.apache.org/jira/browse/HBASE-16649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503964#comment-15503964 ] Matteo Bertozzi commented on HBASE-16649: - let's open another jira to discuss this. since it seems a bit more involved and it is not related directly to DDLs. > Truncate table with splits preserved can cause both data loss and truncated > data appeared again > --- > > Key: HBASE-16649 > URL: https://issues.apache.org/jira/browse/HBASE-16649 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.3 >Reporter: Allan Yang > Attachments: HBASE-16649-v0.patch > > > Since truncate table with splits preserved will delete hfiles and use the > previous regioninfo. It can cause odd behaviors > - Case 1: *Data appeared after truncate* > reproduce procedure： > 1. create a table, let's say 'test' > 2. write data to 'test', make sure memstore of 'test' is not empty > 3. truncate 'test' with splits preserved > 4. kill the regionserver hosting the region(s) of 'test' > 5. start the regionserver, now it is the time to witness the miracle! the > truncated data appeared in table 'test' > - Case 2: *Data loss* > reproduce procedure: > 1. create a table, let's say 'test' > 2. write some data to 'test', no matter how many > 3. truncate 'test' with splits preserved > 4. restart the regionserver to reset the seqid > 5. write some data, but less than 2 since we don't want the seqid to run over > the one in 2 > 6. kill the regionserver hosting the region(s) of 'test' > 7. restart the regionserver. Congratulations! the data writen in 4 is now all > lost > *Why?* > for case 1 > Since preserve splits in truncate table procedure will not change the > regioninfo, when log replay happens, the 'unflushed' data will restore back > to the region > for case 2 > since the flushedSequenceIdByRegion are stored in Master in a map with the > region's encodedName. Although the table is truncated, the region's name is > not changed since we chose to preserve the splits. So after truncate the > table, the region's sequenceid is reset in the regionserver, but not reset in > master. When flush comes and report to master, master will reject the update > of sequenceid since the new one is smaller than the old one. The same happens > in log replay, all the edits writen in 4 will be skipped since they have a > smaller seqid -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[
https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503954#comment-15503954
]
Hadoop QA commented on HBASE-16134:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 42s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 38s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12814579/HBASE-16134_V2.patch |
| JIRA Issue | HBASE-16134 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a22675a2544b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / c5b8aab |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3603/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3603/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3603/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Introduce Cell extension for server side.
> -
>
> Key: HBASE-16134
> URL: https://issues.apache.org/jira/browse/HBASE-16134
> Project: HBase
>
[jira] [Commented] (HBASE-16134) Introduce Cell extension for server side.
[ https://issues.apache.org/jira/browse/HBASE-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503951#comment-15503951 ] Anoop Sam John commented on HBASE-16134: [~saint@gmail.com] Forgot abt this jira for some time.. Coming back to this as am working on subtasks for the parent jira. We do have diff Codec impls which us used at RPC edge. One for writing Tags and one for filtering. We can use those as per the need. Here the method write(OutputStream) is implemented in a Cell impl class. (Like KeyValue). The method contract is to write this Cell bytes in a KV serialization format. Codec just call this method on Cell object and wont do getXXXLength(), getXXXArray() and write individual parts one by one. Means this method impl in Cell objects has to know how to write itself in KV format and what all to write. In case of Codec which needs tags filter, it has to write . Where as when the Codec is the one which need tags the method has to write ..So the point is how to make the method know whether it has to write the tag or not. For this reason only the method was taking a boolean param. Or else how we will do this? We dont want the Codec to call diff getters on Cell and write each part by part. We want to do this in one shot. > Introduce Cell extension for server side. > - > > Key: HBASE-16134 > URL: https://issues.apache.org/jira/browse/HBASE-16134 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16134.patch, HBASE-16134.patch, > HBASE-16134_V2.patch > > > Came after the discussion under HBASE-15721 and HBASE-15879. > InternalCell is a Cell extension. We do have Cell extensions across different > interfaces now. > SettableSeqId > SettableTimestamp > Streamable. > And demand for this keep growing. > So let us include everything into one Interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16335) update to latest apache parent pom
[ https://issues.apache.org/jira/browse/HBASE-16335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503921#comment-15503921 ] Hudson commented on HBASE-16335: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1634 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1634/]) HBASE-16335 RpcClient under heavy load leaks some netty bytebuf (Ram) (ramkrishna: rev c5b8aababe18f65f5db979128a62d8a0686b9dc5) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcConnection.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcConnection.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslWrapHandler.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AbstractRpcClient.java > update to latest apache parent pom > -- > > Key: HBASE-16335 > URL: https://issues.apache.org/jira/browse/HBASE-16335 > Project: HBase > Issue Type: Task > Components: build, dependencies >Reporter: Sean Busbey > Fix For: 2.0.0, 1.4.0 > > > Our apache parent pom is currently ~4 years old. update to latest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiang Li updated HBASE-14882:
-
Status: Patch Available (was: Reopened)
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503868#comment-15503868
]
Xiang Li commented on HBASE-14882:
--
Hi Anoop, I re-uploaded patch 002. It failed when I tried to uploaded it just
now.
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-7612) [JDK8] Replace use of high-scale-lib counters with intrinsic facilities
[ https://issues.apache.org/jira/browse/HBASE-7612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503865#comment-15503865 ] stack commented on HBASE-7612: -- Skimmed. Looks good. Counting is largest consumer of CPU so any improvements here appreciated. > [JDK8] Replace use of high-scale-lib counters with intrinsic facilities > --- > > Key: HBASE-7612 > URL: https://issues.apache.org/jira/browse/HBASE-7612 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Duo Zhang >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-7612-v1.patch, HBASE-7612.patch > > > JEP155 introduces a few new classes (DoubleAccumulator, DoubleAdder, > LongAccumulator, LongAdder) that "internally employ contention-reduction > techniques that provide huge throughput improvements as compared to Atomic > variables". There are applications of these where we are currently using > Cliff Click's high-scale-lib and for metrics. > See http://openjdk.java.net/jeps/155 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
