[jira] [Commented] (HBASE-20621) Unclear error for deleting from not existing table.
[
https://issues.apache.org/jira/browse/HBASE-20621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513377#comment-16513377
]
Mingdao Yang commented on HBASE-20621:
--
Hi everyone!
I'd like to take a stab at fixing it.
> Unclear error for deleting from not existing table.
>
>
> Key: HBASE-20621
> URL: https://issues.apache.org/jira/browse/HBASE-20621
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Priority: Major
>
> When I try to delete a row from a not existing table, the error is quite
> confusing. Instead of getting a table not found exception I got
> {noformat}
> ERROR [main] client.AsyncRequestFutureImpl: Cannot get replica 0 location for
> {"totalColumns":1,"row":"r1","families":{"c1":[{"qualifier":"","vlen":0,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807}
> ERROR: Failed 1 action: t1: 1 time, servers with issues: null
> {noformat}
> That happens, because delete is using AsyncRequestFuture which wraps all
> region location errors into 'Cannot get replica' error. I expect that others
> actions like batch, mutateRow, checkAndDelete behave in the same way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20621) Unclear error for deleting from not existing table.
[
https://issues.apache.org/jira/browse/HBASE-20621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingdao Yang reassigned HBASE-20621:
Assignee: Mingdao Yang
> Unclear error for deleting from not existing table.
>
>
> Key: HBASE-20621
> URL: https://issues.apache.org/jira/browse/HBASE-20621
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Assignee: Mingdao Yang
>Priority: Major
>
> When I try to delete a row from a not existing table, the error is quite
> confusing. Instead of getting a table not found exception I got
> {noformat}
> ERROR [main] client.AsyncRequestFutureImpl: Cannot get replica 0 location for
> {"totalColumns":1,"row":"r1","families":{"c1":[{"qualifier":"","vlen":0,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807}
> ERROR: Failed 1 action: t1: 1 time, servers with issues: null
> {noformat}
> That happens, because delete is using AsyncRequestFuture which wraps all
> region location errors into 'Cannot get replica' error. I expect that others
> actions like batch, mutateRow, checkAndDelete behave in the same way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513369#comment-16513369
]
Hudson commented on HBASE-20733:
Results for branch branch-2
[build #863 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/863/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/863//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/863//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/863//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20575) Fail to config COMPACTION_ENABLED by hbase shell
[
https://issues.apache.org/jira/browse/HBASE-20575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingdao Yang updated HBASE-20575:
-
Attachment: HBASE-20575-branch-1.2.v1.patch
> Fail to config COMPACTION_ENABLED by hbase shell
>
>
> Key: HBASE-20575
> URL: https://issues.apache.org/jira/browse/HBASE-20575
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 1.3.2
>Reporter: Chia-Ping Tsai
>Assignee: Mingdao Yang
>Priority: Major
> Fix For: 1.2.7, 1.3.3
>
> Attachments: HBASE-20575-branch-1.2.patch,
> HBASE-20575-branch-1.2.v1.patch
>
>
> HBASE-19340 backported the missing option from 1.4+ to 1.3 and 1.2. However,
> we made a mistaken to COMPACTION_ENABLED.
> {code:java}
> htd.setCompactionEnabled(JBoolean.valueOf(arg.delete[COMPACTION_ENABLED])) if
> arg[COMPACTION_ENABLED]{code}
> arg.delete[COMPACTION_ENABLED] should be changed to
> arg.delete(COMPACTION_ENABLED)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513360#comment-16513360 ] stack commented on HBASE-20312: --- [~carp84] and [~chancelq] Very nice writeup. Thanks. Left some comments on the doc. This is great stuff. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513351#comment-16513351
]
Hudson commented on HBASE-20733:
Results for branch branch-2.0
[build #429 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/429/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/429//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/429//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/429//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513340#comment-16513340
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #47 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/47/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/47//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/47//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/47//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[
https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513330#comment-16513330
]
stack commented on HBASE-20708:
---
Can I have comment/edit rights on the design doc [~Apache9] please?
bq. Remove RMP just use SCP directly to make meta online. The code in
RMP will be moved to SCP.
This is a new idea of yours?
I think its good.
If a Master joins a cluster where there is no crashed RS, it just scans meta
and away we go?
We'll add a few steps to SCP -- it'll be a little more involved... but should
be ok.
bq. But we still have to wait for meta to be loaded before processing
normal regions in SCP.
We'll still need serverCrashProcessingEnabled type flag to hold up Master
startup until meta is online?
I like this idea too... {{Since a server can only crash once... so queue per
server}}
> Remove the usage of RecoverMetaProcedure in master startup
> --
>
> Key: HBASE-20708
> URL: https://issues.apache.org/jira/browse/HBASE-20708
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch,
> HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch,
> HBASE-20708.patch
>
>
> In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only
> used by RMP to avoid dead lock with MoveRegionProcedure. But we will always
> schedule a RMP when master starting up, so we still need to make sure that
> there is no race between this RMP and other RMPs and SCPs scheduled before
> the master restarts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513327#comment-16513327
]
Hudson commented on HBASE-20733:
Results for branch branch-1.2
[build #365 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/365/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/365//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/365//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/365//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the recovered edits root path
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Summary: WALSplitter uses the rootDir, which is walDir, as the recovered
edits root path (was: WALSplitter uses the rootDir, which is walDir, as the
tableDir root path.)
> WALSplitter uses the rootDir, which is walDir, as the recovered edits root
> path
> ---
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[
https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513315#comment-16513315
]
Hadoop QA commented on HBASE-20708:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
46s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch hbase-protocol-shaded passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} hbase-server: The patch generated 0 new + 342
unchanged - 14 fixed = 342 total (was 356) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
53s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
41s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 25s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 51s{color} |
{color:black} {color} |
\\
\\
[jira] [Updated] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[ https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-20727: --- Attachment: HBASE-20727.004.patch > Persist FlushedSequenceId to speed up WAL split after cluster restart > - > > Key: HBASE-20727 > URL: https://issues.apache.org/jira/browse/HBASE-20727 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch, > HBASE-20727.004.patch, HBASE-20727.patch > > > We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in > ServerManager to record the latest flushed seqids of regions and stores. So > during log split, we can use seqids stored in those maps to filter out the > edits which do not need to be replayed. But, those maps are not persisted. > After cluster restart or master restart, info of flushed seqids are all lost. > Here I offer a way to persist those info to HDFS, even if master restart, we > can still use those info to filter WAL edits and then to speed up replay. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[
https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513310#comment-16513310
]
Allan Yang commented on HBASE-20727:
{quote}
should this be stored in the WAL dir instead of in the root dir?
{quote}
It is not a WAL and it doesn't require 'append' ability when writing this file,
so I think remain in root dir is enough.
{quote}
what happens if the master goes down while writing the file? looks like we'll
get an IOException in loadLastFlushedSequenceIds and act as though the file
doesn't exist?
what happens if the master dies slowly while writing the file and still has it
open when the new master takes over as active? It looks like we will get an
IOException in loadLastFlushedSequenceIds, again when the chore tries in
persistRegionLastFlushedSequenceIds, and eventually just write a new one once
the chore comes around and the lease has expired?
{quote}
Yes, an IOException may be thrown in those cases. But not loading or writing
the file won't cause any problem, it only regress the log split speed to where
we don't have this patch. So no need to deal with those corner cases, just let
it go.
{quote}
How about docs about this addition in the WAL recovery section of the ref guide?
What are the conditions where we'd turn this off? Can we document that as well?
{quote}
Added a comment in the new patch. Normally, we don't have to turn off this
feature. Just provide a switch here.
> Persist FlushedSequenceId to speed up WAL split after cluster restart
> -
>
> Key: HBASE-20727
> URL: https://issues.apache.org/jira/browse/HBASE-20727
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch,
> HBASE-20727.patch
>
>
> We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in
> ServerManager to record the latest flushed seqids of regions and stores. So
> during log split, we can use seqids stored in those maps to filter out the
> edits which do not need to be replayed. But, those maps are not persisted.
> After cluster restart or master restart, info of flushed seqids are all lost.
> Here I offer a way to persist those info to HDFS, even if master restart, we
> can still use those info to filter WAL edits and then to speed up replay.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20736) Fix smattering of error-prone warnings
[
https://issues.apache.org/jira/browse/HBASE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513303#comment-16513303
]
Hadoop QA commented on HBASE-20736:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hbase-build-configuration {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
17s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
17s{color} | {color:red} hbase-zookeeper in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
22s{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
8s{color} | {color:green} hbase-build-configuration in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} hbase-common generated 0 new + 37 unchanged - 5
fixed = 37 total (was 42) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} hbase-hadoop-compat generated 0 new + 3 unchanged -
1 fixed = 3 total (was 4) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 16s{color}
| {color:red} hbase-hadoop2-compat generated 1 new + 33 unchanged - 6 fixed =
34 total (was 39) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red} hbase-client generated 1 new + 97 unchanged - 6 fixed = 98 total
(was 103) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} hbase-zookeeper in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} hbase-http generated 0 new + 16 unchanged - 1 fixed
= 16 total (was 17) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} hbase-procedure generated 0 new + 4 unchanged - 5
fixed = 4 total (was 9) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 22s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} hbase-common: The patch generated 3 new + 12 unchanged
- 2 fixed = 15 total (was 14) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
27s{color} | {color:red} hbase-client: The patch generated 1 new + 54 unchanged
- 0 fixed = 55 total (was 54) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hbase-http: The patch generated 2 new + 0 unchanged -
0 fixed = 2 total (was 0) {color} |
|
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Priority: Critical (was: Major)
Component/s: (was: hbase)
wal
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513295#comment-16513295 ] Duo Zhang commented on HBASE-20708: --- Fix master failover, and add more comments. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20708: -- Attachment: HBASE-20708-v5.patch > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708-v5.patch, > HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513286#comment-16513286
]
Sean Busbey commented on HBASE-20723:
-
could someone update the component and severity? It sounds like "Recovery"
would be a good component and the severity should be either Critical or Blocker?
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513284#comment-16513284
]
Hadoop QA commented on HBASE-20723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
58s{color} | {color:red} hbase-server: The patch generated 1 new + 75 unchanged
- 2 fixed = 76 total (was 77) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 2s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}108m
8s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20723 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927905/20723.v9.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 17178f5d60dd 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[
https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513282#comment-16513282
]
Sean Busbey commented on HBASE-20727:
-
should this be stored in the WAL dir instead of in the root dir?
what happens if the master goes down while writing the file? looks like we'll
get an IOException in {{loadLastFlushedSequenceIds}} and act as though the file
doesn't exist?
what happens if the master dies slowly while writing the file and still has it
open when the new master takes over as active? It looks like we will get an
IOException in {{loadLastFlushedSequenceIds}}, again when the chore tries in
{{persistRegionLastFlushedSequenceIds}}, and eventually just write a new one
once the chore comes around and the lease has expired?
How about docs about this addition in the WAL recovery section of the ref guide?
{code}
+ public static final String PERSIST_FLUSHEDSEQUENCEID =
+ "hbase.master.persist.flushedsequenceid.enabled";
{code}
What are the conditions where we'd turn this off? Can we document that as well?
> Persist FlushedSequenceId to speed up WAL split after cluster restart
> -
>
> Key: HBASE-20727
> URL: https://issues.apache.org/jira/browse/HBASE-20727
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch,
> HBASE-20727.patch
>
>
> We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in
> ServerManager to record the latest flushed seqids of regions and stores. So
> during log split, we can use seqids stored in those maps to filter out the
> edits which do not need to be replayed. But, those maps are not persisted.
> After cluster restart or master restart, info of flushed seqids are all lost.
> Here I offer a way to persist those info to HDFS, even if master restart, we
> can still use those info to filter WAL edits and then to speed up replay.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20695) Implement table level RegionServer replication metrics
[ https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20695: --- Fix Version/s: 2.1.0 > Implement table level RegionServer replication metrics > --- > > Key: HBASE-20695 > URL: https://issues.apache.org/jira/browse/HBASE-20695 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20695.master.001.patch, > HBASE-20695.master.002.patch, HBASE-20695.master.003.patch, > HBASE-20695.master.004.patch, HBASE-20695.master.005.patch, > HBASE-20695.master.006.patch, HBASE-20695.master.007.patch, > HBASE-20695.master.008.patch, HBASE-20695.master.009.patch, > HBASE-20695.master.010.patch > > > Region server metrics now are mainly global metrics. It would be nice to have > table level metrics such as table level source.AgeOfLastShippedOp to indicate > operators which table's replication is lagging behind. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20695) Implement table level RegionServer replication metrics
[ https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513280#comment-16513280 ] Guanghao Zhang edited comment on HBASE-20695 at 6/15/18 3:20 AM: - Pushed to master and branch-2. Thanks [~xucang] for contributing. And please use the jira id and jira title as the commit message in the future. Thanks. :-) was (Author: zghaobac): Pushed to master and branch-2. Thanks [~xucang] for contributing. And please use the jira id and jira title as the commit message. Thanks. :-) > Implement table level RegionServer replication metrics > --- > > Key: HBASE-20695 > URL: https://issues.apache.org/jira/browse/HBASE-20695 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20695.master.001.patch, > HBASE-20695.master.002.patch, HBASE-20695.master.003.patch, > HBASE-20695.master.004.patch, HBASE-20695.master.005.patch, > HBASE-20695.master.006.patch, HBASE-20695.master.007.patch, > HBASE-20695.master.008.patch, HBASE-20695.master.009.patch, > HBASE-20695.master.010.patch > > > Region server metrics now are mainly global metrics. It would be nice to have > table level metrics such as table level source.AgeOfLastShippedOp to indicate > operators which table's replication is lagging behind. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20695) Implement table level RegionServer replication metrics
[ https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20695: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master and branch-2. Thanks [~xucang] for contributing. And please use the jira id and jira title as the commit message. Thanks. :-) > Implement table level RegionServer replication metrics > --- > > Key: HBASE-20695 > URL: https://issues.apache.org/jira/browse/HBASE-20695 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Attachments: HBASE-20695.master.001.patch, > HBASE-20695.master.002.patch, HBASE-20695.master.003.patch, > HBASE-20695.master.004.patch, HBASE-20695.master.005.patch, > HBASE-20695.master.006.patch, HBASE-20695.master.007.patch, > HBASE-20695.master.008.patch, HBASE-20695.master.009.patch, > HBASE-20695.master.010.patch > > > Region server metrics now are mainly global metrics. It would be nice to have > table level metrics such as table level source.AgeOfLastShippedOp to indicate > operators which table's replication is lagging behind. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[ https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513269#comment-16513269 ] Sean Busbey commented on HBASE-20727: - please fix the checkstyle issues pointed out by qabot. > Persist FlushedSequenceId to speed up WAL split after cluster restart > - > > Key: HBASE-20727 > URL: https://issues.apache.org/jira/browse/HBASE-20727 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch, > HBASE-20727.patch > > > We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in > ServerManager to record the latest flushed seqids of regions and stores. So > during log split, we can use seqids stored in those maps to filter out the > edits which do not need to be replayed. But, those maps are not persisted. > After cluster restart or master restart, info of flushed seqids are all lost. > Here I offer a way to persist those info to HDFS, even if master restart, we > can still use those info to filter WAL edits and then to speed up replay. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20736) Fix smattering of error-prone warnings
[ https://issues.apache.org/jira/browse/HBASE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20736: -- Status: Patch Available (was: Open) > Fix smattering of error-prone warnings > -- > > Key: HBASE-20736 > URL: https://issues.apache.org/jira/browse/HBASE-20736 > Project: HBase > Issue Type: Bug >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20736.master.001.patch, > HBASE-20736.master.002.patch > > > I wanted some coding on cruise control for a few hours tonight, so I deiced > to fix error-prone warnings. I only fixed ones that were straight-forward, > easy to do, and could defensibly have value in the fix. I skipped a lot, and > won't claim to have any particular reason. Got as far as hbase-server in the > output before deciding to move on with my life (didn't cover the tests at > all). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20736) Fix smattering of error-prone warnings
[ https://issues.apache.org/jira/browse/HBASE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20736: -- Attachment: HBASE-20736.master.002.patch > Fix smattering of error-prone warnings > -- > > Key: HBASE-20736 > URL: https://issues.apache.org/jira/browse/HBASE-20736 > Project: HBase > Issue Type: Bug >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20736.master.001.patch, > HBASE-20736.master.002.patch > > > I wanted some coding on cruise control for a few hours tonight, so I deiced > to fix error-prone warnings. I only fixed ones that were straight-forward, > easy to do, and could defensibly have value in the fix. I skipped a lot, and > won't claim to have any particular reason. Got as far as hbase-server in the > output before deciding to move on with my life (didn't cover the tests at > all). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20736) Fix smattering of error-prone warnings
[ https://issues.apache.org/jira/browse/HBASE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20736: -- Attachment: HBASE-20736.master.002.patch > Fix smattering of error-prone warnings > -- > > Key: HBASE-20736 > URL: https://issues.apache.org/jira/browse/HBASE-20736 > Project: HBase > Issue Type: Bug >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20736.master.001.patch > > > I wanted some coding on cruise control for a few hours tonight, so I deiced > to fix error-prone warnings. I only fixed ones that were straight-forward, > easy to do, and could defensibly have value in the fix. I skipped a lot, and > won't claim to have any particular reason. Got as far as hbase-server in the > output before deciding to move on with my life (didn't cover the tests at > all). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20736) Fix smattering of error-prone warnings
[ https://issues.apache.org/jira/browse/HBASE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20736: -- Attachment: (was: HBASE-20736.master.002.patch) > Fix smattering of error-prone warnings > -- > > Key: HBASE-20736 > URL: https://issues.apache.org/jira/browse/HBASE-20736 > Project: HBase > Issue Type: Bug >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20736.master.001.patch > > > I wanted some coding on cruise control for a few hours tonight, so I deiced > to fix error-prone warnings. I only fixed ones that were straight-forward, > easy to do, and could defensibly have value in the fix. I skipped a lot, and > won't claim to have any particular reason. Got as far as hbase-server in the > output before deciding to move on with my life (didn't cover the tests at > all). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20736) Fix smattering of error-prone warnings
[ https://issues.apache.org/jira/browse/HBASE-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20736: -- Attachment: HBASE-20736.master.001.patch > Fix smattering of error-prone warnings > -- > > Key: HBASE-20736 > URL: https://issues.apache.org/jira/browse/HBASE-20736 > Project: HBase > Issue Type: Bug >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20736.master.001.patch > > > I wanted some coding on cruise control for a few hours tonight, so I deiced > to fix error-prone warnings. I only fixed ones that were straight-forward, > easy to do, and could defensibly have value in the fix. I skipped a lot, and > won't claim to have any particular reason. Got as far as hbase-server in the > output before deciding to move on with my life (didn't cover the tests at > all). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20736) Fix smattering of error-prone warnings
Mike Drob created HBASE-20736: - Summary: Fix smattering of error-prone warnings Key: HBASE-20736 URL: https://issues.apache.org/jira/browse/HBASE-20736 Project: HBase Issue Type: Bug Reporter: Mike Drob Assignee: Mike Drob I wanted some coding on cruise control for a few hours tonight, so I deiced to fix error-prone warnings. I only fixed ones that were straight-forward, easy to do, and could defensibly have value in the fix. I skipped a lot, and won't claim to have any particular reason. Got as far as hbase-server in the output before deciding to move on with my life (didn't cover the tests at all). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513245#comment-16513245
]
Hudson commented on HBASE-20733:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #423 (See
[https://builds.apache.org/job/HBase-1.3-IT/423/])
HBASE-20733 QABot should run checkstyle tests if the checkstyle configs
(busbey: rev b96cf3da5c2253e2183cf31a2b20e0ab22d1dabd)
* (edit) dev-support/hbase-personality.sh
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513244#comment-16513244
]
Hudson commented on HBASE-20733:
SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1124 (See
[https://builds.apache.org/job/HBase-1.2-IT/1124/])
HBASE-20733 QABot should run checkstyle tests if the checkstyle configs
(busbey: rev 20772f139d7d60ecbd9228f007c35547fa64a196)
* (edit) dev-support/hbase-personality.sh
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20332:
Attachment: HBASE-20332.5.patch
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513236#comment-16513236
]
Sean Busbey commented on HBASE-20332:
-
v5
- rebased onto master to pick up the changes in HBASE-20733
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20733:
Resolution: Fixed
Fix Version/s: 2.0.2
1.4.6
1.3.3
1.2.7
1.5.0
2.1.0
3.0.0
Status: Resolved (was: Patch Available)
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.2.7, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[ https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513229#comment-16513229 ] Allan Yang commented on HBASE-20727: Does anyone else has some advice? I will commit to master branch if no objection. > Persist FlushedSequenceId to speed up WAL split after cluster restart > - > > Key: HBASE-20727 > URL: https://issues.apache.org/jira/browse/HBASE-20727 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch, > HBASE-20727.patch > > > We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in > ServerManager to record the latest flushed seqids of regions and stores. So > during log split, we can use seqids stored in those maps to filter out the > edits which do not need to be replayed. But, those maps are not persisted. > After cluster restart or master restart, info of flushed seqids are all lost. > Here I offer a way to persist those info to HDFS, even if master restart, we > can still use those info to filter WAL edits and then to speed up replay. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513224#comment-16513224
]
Hadoop QA commented on HBASE-20723:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} hbase-server: The patch generated 0 new + 76
unchanged - 1 fixed = 76 total (was 77) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}107m
23s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20723 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927884/20723.v8.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3cf0cbe92979 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13258/testReport/ |
| Max.
[jira] [Updated] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20733:
Priority: Minor (was: Major)
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513212#comment-16513212
]
Hadoop QA commented on HBASE-20733:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-20733 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} HBASE-20733 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
26s{color} | {color:green} HBASE-20733 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} HBASE-20733 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
10s{color} | {color:green} hbase-checkstyle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927907/HBASE-20733-HBASE-20733.1.patch
|
| Optional Tests | asflicense checkstyle javac javadoc unit xml |
| uname | Linux 65dcdd66fafc 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-20733 / 73366c8621 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13261/testReport/ |
| Max. process+thread count | 83 (vs. ulimit of 1) |
| modules | C: hbase-checkstyle U: hbase-checkstyle |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13261/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513211#comment-16513211
]
Hadoop QA commented on HBASE-20733:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-20733 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} HBASE-20733 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
51s{color} | {color:green} HBASE-20733 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} HBASE-20733 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
8s{color} | {color:green} hbase-checkstyle in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927907/HBASE-20733-HBASE-20733.1.patch
|
| Optional Tests | asflicense checkstyle javac javadoc unit xml |
| uname | Linux 38a25549a727 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-20733 / 73366c8621 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13263/testReport/ |
| Max. process+thread count | 93 (vs. ulimit of 1) |
| modules | C: hbase-checkstyle U: hbase-checkstyle |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13263/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513208#comment-16513208
]
Hadoop QA commented on HBASE-20723:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
1s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} hbase-server: The patch generated 0 new + 76
unchanged - 1 fixed = 76 total (was 77) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 54s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}158m 22s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}194m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestClientOperationTimeout |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20723 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927880/20723.v7.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 24c2d8f06e1f 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513205#comment-16513205 ] Ted Yu commented on HBASE-20734: After HBASE-17437 was incorporated into 1.4 release, the recovered edits dir can diverge from where wal.dir is configured. HBASE-20723 fixes data loss. This issue is to get recovered edits dir colocated with wal.dir, considering combinations users may already have in their deployment. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513204#comment-16513204 ] Duo Zhang commented on HBASE-20708: --- It seems that the master session expire still can not pass, will dig more. Let's see if there are other problems. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20708: -- Attachment: HBASE-20708-v4.patch > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708-v4.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513200#comment-16513200 ] Duo Zhang commented on HBASE-20708: --- OK seems just a bug, missed a negative... Let me post a new patch. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513198#comment-16513198
]
Mike Drob commented on HBASE-20733:
---
+1 looks fine, don't need to wait for a branch and nightly
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20733:
Attachment: HBASE-20733-HBASE-20733.1.patch
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733-HBASE-20733.1.patch, HBASE-20733.0.patch,
> HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513193#comment-16513193
]
Hadoop QA commented on HBASE-20733:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} The patch generated 0 new + 0 unchanged - 2 fixed =
0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927906/HBASE-20733.1.patch |
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux a35dac831b02 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| shellcheck | v0.4.4 |
| Max. process+thread count | 42 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13260/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch, HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513192#comment-16513192
]
Hadoop QA commented on HBASE-20733:
---
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HBASE-Build/13260/console in case of
problems.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch, HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20733:
Status: Patch Available (was: In Progress)
v1
- do checking for checkstyle config changes in personality_file_tests
- move custom mvnsite check there while we're at it, since the old way relies
on undefined yetus behavior
- fix a couple of unrelated shellcheck complaints
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch, HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513191#comment-16513191
]
Sean Busbey commented on HBASE-20733:
-
okay fun finding: changes to our personality cause the "Re-executing against
the patched versions to perform further tests." message above, but don't appear
to be reflected in what actually runs.
I tested this version locally by applying the v1 commit and then testing with
the WIP commit from before. I guess I can get a precommit run by using a branch.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch, HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20733:
Attachment: HBASE-20733.1.patch
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch, HBASE-20733.1.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-12806) [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck
[ https://issues.apache.org/jira/browse/HBASE-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-12806: -- Resolution: Duplicate Status: Resolved (was: Patch Available) HBASE-14570 made it so all creates got through a utility method and wait for table online. > [hbck] move admin.create() to HBaseTestingUtility.createTable in TestHBaseFsck > -- > > Key: HBASE-12806 > URL: https://issues.apache.org/jira/browse/HBASE-12806 > Project: HBase > Issue Type: Bug > Components: hbck >Affects Versions: 1.0.0, 2.0.0 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Minor > Attachments: > 0001-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch, > 0002-HBASE-12806-hbck-move-admin.create-to-HBaseTestingUt.patch > > > TestHBaseFsck should wait until all regions have been assigned after the > table has been created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513182#comment-16513182 ] huaxiang sun edited comment on HBASE-20710 at 6/15/18 12:45 AM: Yeah, for the table.put path, we can save copyFamily as it has been already copied out. For table.batch case (cellblock), this copy is unavoidable. I have attached the allocation profiles for the put case (netty rpc server), with the patch, the cloneFamily() disappears in the allocation profile. It was run with PE randomWrite. Minor bonus, with the same family buffer, key compare for search in TreeMap is a shortcircuit, it does not compare byte to byte. was (Author: huaxiang): Yeah, for the table.put path, we can save copyFamily as it has been already copied out. For table.batch case (cellblock), this copy is unavoidable. I have attached the allocation profiles for the put case (netty rpc server), with the patch, the cloneFamily() disappears in the allocation profile. It was run with PE randomWrite. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20710) extra cloneFamily() in Mutation.add(Cell)
[ https://issues.apache.org/jira/browse/HBASE-20710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513182#comment-16513182 ] huaxiang sun commented on HBASE-20710: -- Yeah, for the table.put path, we can save copyFamily as it has been already copied out. For table.batch case (cellblock), this copy is unavoidable. I have attached the allocation profiles for the put case (netty rpc server), with the patch, the cloneFamily() disappears in the allocation profile. It was run with PE randomWrite. > extra cloneFamily() in Mutation.add(Cell) > - > > Key: HBASE-20710 > URL: https://issues.apache.org/jira/browse/HBASE-20710 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.1 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.1 > > Attachments: HBASE-20710-master-v001.patch, > HBASE-20710-master-v002.patch, alloc-put-fix.svg, alloc-put-orig.svg > > > The cpu profiling shows that during PE randomWrite testing, about 1 percent > of time is spent in cloneFamily. Reviewing code found that when a cell is DBB > backed ByteBuffKeyValueCell (which is default with Netty Rpc), > cell.getFamilyArray() will call cloneFamily() and there is again a > cloneFamily() in the following line of the code. since this is the critical > write path processing, this needs to be optimized. > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L791 > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java#L795 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513181#comment-16513181 ] Duo Zhang commented on HBASE-20708: --- OK, there is a problem that how we deal with master graceful shutdown. In this case, the new design will be broken as it will not schedule any SCP so no region will be online. Let me dig more. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-9083) Downstreamers have to include a load of runtime dependencies
[
https://issues.apache.org/jira/browse/HBASE-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated HBASE-9083:
-
Resolution: Done
Assignee: (was: stack)
Status: Resolved (was: Patch Available)
> Downstreamers have to include a load of runtime dependencies
>
>
> Key: HBASE-9083
> URL: https://issues.apache.org/jira/browse/HBASE-9083
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: stack
>Priority: Critical
> Labels: beginner
> Attachments: HBASE_9083.patch
>
>
> Here is example from 0.95. Downstream project includes hbase-client ONLY.
> To run the downstream project, here are the runtime dependencies currently.
> This is hadoop1.
> {code}
> java -cp
> target/client-1.0-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/hbase/hbase-client/0.95.2-hadoop1-SNAPSHOT/hbase-client-0.95.2-hadoop1-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/hbase/hbase-common/0.95.2-hadoop1-SNAPSHOT/hbase-common-0.95.2-hadoop1-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar:/Users/stack/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/stack/.m2/repository/com/google/protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar:/Users/stack/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/stack/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/stack/.m2/repository/org/apache/hbase/hbase-protocol/0.95.2-hadoop1-SNAPSHOT/hbase-protocol-0.95.2-hadoop1-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/stack/.m2/repository/org/slf4j/slf4j-api/1.6.4/slf4j-api-1.6.4.jar:/Users/stack/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar:/Users/stack/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/Users/stack/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/Users/stack/.m2/repository/org/cloudera/htrace/htrace/1.50/htrace-1.50.jar:/Users/stack/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar:/Users/stack/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar
> org.hbase.downstream.Client
> {code}
> Thats:
> {code}
> hbase-client
> base-common
> hbase-protocol
> hadoop-core
> commons-logging
> protobuf
> commons-lang
> commons-configuration
> zookeeper
> slf4j-api (AND commons-logging!)
> guava
> jackson-mapper-asl
> jackson-core-asl
> htrace
> slf4j-log4j12
> slf4j
> {code}
> Most of the above come in because of hadoop and zk (zk wants slf4j).
> Can we shed any of these?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9083) Downstreamers have to include a load of runtime dependencies
[
https://issues.apache.org/jira/browse/HBASE-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513180#comment-16513180
]
Mike Drob commented on HBASE-9083:
--
Yea, obviated by HBASE-13517 and further by HBASE-20331.
> Downstreamers have to include a load of runtime dependencies
>
>
> Key: HBASE-9083
> URL: https://issues.apache.org/jira/browse/HBASE-9083
> Project: HBase
> Issue Type: Sub-task
> Components: build
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Labels: beginner
> Attachments: HBASE_9083.patch
>
>
> Here is example from 0.95. Downstream project includes hbase-client ONLY.
> To run the downstream project, here are the runtime dependencies currently.
> This is hadoop1.
> {code}
> java -cp
> target/client-1.0-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/hbase/hbase-client/0.95.2-hadoop1-SNAPSHOT/hbase-client-0.95.2-hadoop1-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/hbase/hbase-common/0.95.2-hadoop1-SNAPSHOT/hbase-common-0.95.2-hadoop1-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar:/Users/stack/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/stack/.m2/repository/com/google/protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar:/Users/stack/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/stack/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/stack/.m2/repository/org/apache/hbase/hbase-protocol/0.95.2-hadoop1-SNAPSHOT/hbase-protocol-0.95.2-hadoop1-SNAPSHOT.jar:/Users/stack/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/Users/stack/.m2/repository/org/slf4j/slf4j-api/1.6.4/slf4j-api-1.6.4.jar:/Users/stack/.m2/repository/com/google/guava/guava/12.0.1/guava-12.0.1.jar:/Users/stack/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/Users/stack/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/Users/stack/.m2/repository/org/cloudera/htrace/htrace/1.50/htrace-1.50.jar:/Users/stack/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar:/Users/stack/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar
> org.hbase.downstream.Client
> {code}
> Thats:
> {code}
> hbase-client
> base-common
> hbase-protocol
> hadoop-core
> commons-logging
> protobuf
> commons-lang
> commons-configuration
> zookeeper
> slf4j-api (AND commons-logging!)
> guava
> jackson-mapper-asl
> jackson-core-asl
> htrace
> slf4j-log4j12
> slf4j
> {code}
> Most of the above come in because of hadoop and zk (zk wants slf4j).
> Can we shed any of these?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513177#comment-16513177
]
Hadoop QA commented on HBASE-20733:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
4s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} The patch generated 0 new + 0 unchanged - 2 fixed =
0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}188m
18s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}203m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927860/HBASE-20733.0.patch |
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
xml |
| uname | Linux 9c35556985ee 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13255/testReport/ |
| Max. process+thread count | 4866 (vs. ulimit of 1) |
| modules | C: hbase-checkstyle . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13255/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.v9.txt
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: (was: 20723.v9.txt)
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513166#comment-16513166 ] Duo Zhang commented on HBASE-20708: --- OK, not too many. Let me check them. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20708-v1.patch, HBASE-20708-v2.patch, > HBASE-20708-v3.patch, HBASE-20708.patch > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513169#comment-16513169 ] Zach York commented on HBASE-20734: --- Before HBASE-20723 goes in, there is no chance of this happening, right? Currently it will fail if hbase.wal.dir is set to anything but the default. We could remove the headache of BC if we fixed this right the first time. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513150#comment-16513150 ] Ted Yu commented on HBASE-20734: Suppose the new region server only knows about the recovered edits dir under wal.dir, it may miss edits located under recovered edits dir under root dir. So the new code needs to look for edits in both places. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513139#comment-16513139
]
Ted Yu commented on HBASE-20723:
Patch v9 removed the FileSystem parameter to getRegionSplitEditsPath() since it
can be obtained thru the conf.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.v9.txt
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt,
> 20723.v9.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513113#comment-16513113
]
Zach York commented on HBASE-20723:
---
Removed 1.1.2 from affects since the backport isn't in the public repo. This
affects 1.4.0+
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zach York updated HBASE-20723:
--
Affects Version/s: (was: 1.1.2)
1.4.0
1.4.1
1.4.2
1.4.3
1.4.4
2.0.0
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513107#comment-16513107
]
Zach York commented on HBASE-20723:
---
[~yuzhih...@gmail.com] The patch looks good to me. Let's see what QA says.
[~elserj] Do you think this is enough to reroll a 1.4.5 RC? It isn't a default
config, but still quite serious for those that set this config.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513097#comment-16513097 ] Zach York commented on HBASE-20734: --- doesn't split log only run before region opening so shouldn't rolling upgrade work? Or is there a case where the log is split, not applied, and tries to split/recover again? > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513077#comment-16513077 ] Ted Yu commented on HBASE-20734: Moving recovered edits dir over to under wal.dir is the obvious action to take. But we should consider effect on rolling upgrade, etc. The new code needs to be able to recognize both locations. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513075#comment-16513075
]
Ted Yu commented on HBASE-20723:
Patch v8 does variable renaming w.r.t. walDir.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.v8.txt
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513073#comment-16513073
]
Josh Elser commented on HBASE-20681:
{quote}So the end result here is just that we get hamcrest in the lib/
directory with everything else?
{quote}
Yup yup!
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513067#comment-16513067
]
Zach York commented on HBASE-20723:
---
Okay, let's do the quick fix here and look to HBASE-20734 for the longer term
solution. In regards to your existing code, can you change all other
occurrences of rootDir (where it means walDir) to walDir to avoid confusion in
the mean time?
Let me start working on seeing if moving recovered edits to the WAL dir fixes
HBASE-20734.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513062#comment-16513062
]
Ted Yu commented on HBASE-20723:
bq. some indication of the number of records replayed
The above metric alone may not be enough to indicate whether there was data
loss. I suggest formulating the combination of factors for alerting the user in
another issue.
For the last paragraph, I agree.
I already logged a JIRA - HBASE-20734
For this data loss bug, I suggest we fix in this issue since the proper
solution to HBASE-20734 may take longer to come up.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18304) Start enforcing upperbounds on dependencies
[
https://issues.apache.org/jira/browse/HBASE-18304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513055#comment-16513055
]
Hadoop QA commented on HBASE-18304:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HBASE-18304 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-18304 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880263/HBASE-18304.master.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13257/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Start enforcing upperbounds on dependencies
> ---
>
> Key: HBASE-18304
> URL: https://issues.apache.org/jira/browse/HBASE-18304
> Project: HBase
> Issue Type: Task
> Components: build, dependencies
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Tamas Penzes
>Priority: Major
> Labels: beginner
> Attachments: HBASE-18304.master.001.patch,
> HBASE-18304.master.002.patch, HBASE-18304.master.002.patch,
> HBASE-18304.master.003.patch
>
>
> would be nice to get this going before our next major version.
> http://maven.apache.org/enforcer/enforcer-rules/requireUpperBoundDeps.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513049#comment-16513049
]
Sean Busbey commented on HBASE-20681:
-
oh I totally missed that there was still a change after "add junit / hamcrest
to the classpath for the integration test" thing.
So the end result here is just that we get hamcrest in the lib/ directory with
everything else?
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513048#comment-16513048
]
Zach York commented on HBASE-20723:
---
It's great that we have these JMX values, but unless an admin happens to look
at these, there isn't anything calling out a potential issue. If we log the
replay_num_ops for each recovery attempt, then maybe it would be useful.
{quote}Once this logic error is fixed, I am not aware of other scenario where
the message should be WARN.
{quote}
If only software development were that simple :)... There is no guarantee that
this can't break again (obviously we will do our best) or a different edge case
will break something like this. Unless there is a way to check with 100%
certainty that this is expected behavior, this log line is still useful. Though
I would feel better about leaving it at info if it were possible to see from
some other log line that things might not be operating correctly. It seems too
many assumptions are being made about the recovered edits directory.
Related to the actual change...
I've been thinking about this a bit and why does it make sense for WALs to be
under hbase.wal.dir, but for recovered.edits (basically the split log) to be
under the root directory. It seems to me that both should be under the
hbase.wal.dir.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513043#comment-16513043
]
Ted Yu commented on HBASE-20723:
Patch v7 sets wal.dir to different location than rootdir in TestWALFactory
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20723:
---
Attachment: 20723.v7.txt
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18304) Start enforcing upperbounds on dependencies
[ https://issues.apache.org/jira/browse/HBASE-18304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513039#comment-16513039 ] Mike Drob commented on HBASE-18304: --- I tried to pick this up again and so far I've been running into several dependencies where we don't depend on it at all, but enforcer plugin complains because it turns out hadoop is internally inconsistent (usually between hadoop-client and hadoop-minicluster having transitive dependancies on things). This might be a bug in the enforcer plugin, or we should push hadoop to be cleaner before we start trying to enforce things on our own side. > Start enforcing upperbounds on dependencies > --- > > Key: HBASE-18304 > URL: https://issues.apache.org/jira/browse/HBASE-18304 > Project: HBase > Issue Type: Task > Components: build, dependencies >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Tamas Penzes >Priority: Major > Labels: beginner > Attachments: HBASE-18304.master.001.patch, > HBASE-18304.master.002.patch, HBASE-18304.master.002.patch, > HBASE-18304.master.003.patch > > > would be nice to get this going before our next major version. > http://maven.apache.org/enforcer/enforcer-rules/requireUpperBoundDeps.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513029#comment-16513029
]
Sean Busbey commented on HBASE-20733:
-
changes to {{src/main}} within a module will trigger a unit test run, so
they're running because we keep our checkstyle config in {{src/main/resources}}
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[
https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513028#comment-16513028
]
Hadoop QA commented on HBASE-20695:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 8s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
44s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20695 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927864/HBASE-20695.master.010.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 17826de211af 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13253/testReport/ |
| Max. process+thread count | 4189 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13253/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Implement table level
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513023#comment-16513023
]
Ted Yu commented on HBASE-20723:
For #2 above, we already have JMX for :
{code}
"Replay_num_ops" : 0,
"Replay_min" : 0,
"Replay_max" : 0,
"Replay_mean" : 0.0,
"Replay_median" : 0.0,
"Replay_75th_percentile" : 0.0,
"Replay_95th_percentile" : 0.0,
"Replay_99th_percentile" : 0.0,
{code}
bq. very least be a WARN to indicate something potentially wrong happened
Once this logic error is fixed, I am not aware of other scenario where the
message should be WARN.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513019#comment-16513019
]
Zach York commented on HBASE-20723:
---
[~yuzhih...@gmail.com] I did reproduce the issue on my side as well. Let me
review your patch as well.
I think going forward we need two things to prevent something like this
happening again:
# Tests that utilize hbase.wal.dir (on a different FS and path) to validate
that edits are able to be replayed and logs are split from a user level (put,
kill RS, restart RS, check to ensure edit is present in table).
# Improve on this log messaging around here. There should be some indication
of the number of records replayed or something as the current logging is easy
to miss... Considering this log means that edits won't be applied for that
region, this should at the very least be a WARN to indicate something
potentially wrong happened.
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512999#comment-16512999
]
Sean Busbey commented on HBASE-20733:
-
okay no checkstyle is because I need to use Yetus' "personality file testing"
instead of trying to directly override how the checkstyle plugin checks for
files.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-20733:
Status: In Progress (was: Patch Available)
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512993#comment-16512993
]
Hadoop QA commented on HBASE-20733:
---
(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at
https://builds.apache.org/job/PreCommit-HBASE-Build/13255/console in case of
problems.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512992#comment-16512992
]
Sean Busbey commented on HBASE-20733:
-
rerunning with debug.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512991#comment-16512991
]
Sean Busbey commented on HBASE-20733:
-
no tests because the test is the patch here.
unit test failures don't look related. I'm not sure why these ran since this
change wasn't to any files that ought to cause a unit test run.
also no checkstyle, what.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do checkstyle tests when java files are altered. we should
> also run if our checkstyle configs in {{hbase-checkstyle}} are altered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20561) The way we stop a ReplicationSource may cause the RS down
[
https://issues.apache.org/jira/browse/HBASE-20561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512980#comment-16512980
]
Hudson commented on HBASE-20561:
Results for branch master
[build #365 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/365/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> The way we stop a ReplicationSource may cause the RS down
> -
>
> Key: HBASE-20561
> URL: https://issues.apache.org/jira/browse/HBASE-20561
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-20561.master.001.patch,
> HBASE-20561.master.002.patch, HBASE-20561.master.003.patch,
> HBASE-20561.master.004.patch, HBASE-20561.master.005.patch
>
>
> See this:
> https://builds.apache.org/job/HBASE-Flaky-Tests/31125/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL-output.txt
> {noformat}
> 2018-05-09 15:07:00,887 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.RefreshPeerCallable(52): Received a peer change event, peerId=2,
> type=REMOVE_PEER
> 2018-05-09 15:07:00,890 INFO [RS_REFRESH_PEER-regionserver/asf916:0-1]
> regionserver.ReplicationSource(485): Closing source
> 2-asf916.gq1.ygridcore.net,36287,1525878368395 because: Replication stream
> was removed by a user
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-0,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617): regionserver:34308-0x163456ff2490004,
> quorum=localhost:60149, baseZNode=/1 Received InterruptedException, will
> interrupt current thread and rethrow a SystemErrorException
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:871)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231)
> at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationQueueStorage.removeWAL(ZKReplicationQueueStorage.java:198)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.lambda$cleanOldLogs$8(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.abortWhenFail(ReplicationSourceManager.java:454)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:526)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.cleanOldLogs(ReplicationSourceManager.java:506)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.logPositionAndCleanOldLogs(ReplicationSourceManager.java:489)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.updateLogPosition(ReplicationSourceShipper.java:231)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.shipEdits(ReplicationSourceShipper.java:133)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceShipper.run(ReplicationSourceShipper.java:103)
> 2018-05-09 15:07:00,892 DEBUG
> [ReplicationExecutor-0.replicationSource,2-asf916.gq1.ygridcore.net,36287,1525878368395.replicationSource.shipperasf916.gq1.ygridcore.net%2C36287%2C1525878368395.asf916.gq1.ygridcore.net%2C36287%2C1525878368395.regiongroup-1,2-asf916.gq1.ygridcore.net,36287,1525878368395]
> zookeeper.ZKWatcher(617):
[jira] [Commented] (HBASE-20630) B: Delete command enhancements
[
https://issues.apache.org/jira/browse/HBASE-20630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512982#comment-16512982
]
Hudson commented on HBASE-20630:
Results for branch master
[build #365 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/365/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> B: Delete command enhancements
>
>
> Key: HBASE-20630
> URL: https://issues.apache.org/jira/browse/HBASE-20630
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20630-v1.patch, HBASE-20630-v2.patch,
> HBASE-20630-v3.patch, HBASE-20630-v4.patch
>
>
> Make the command more useable. Currently, user needs to provide list of
> backup ids to delete. It would be nice to have more convenient options, such
> as: deleting all backups which are older than XXX days, etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20625) refactor some WALCellCodec related code
[
https://issues.apache.org/jira/browse/HBASE-20625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512983#comment-16512983
]
Hudson commented on HBASE-20625:
Results for branch master
[build #365 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/365/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> refactor some WALCellCodec related code
> ---
>
> Key: HBASE-20625
> URL: https://issues.apache.org/jira/browse/HBASE-20625
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20625.branch-2.001.patch,
> HBASE-20625.master.001.patch, HBASE-20625.master.002.patch,
> HBASE-20625.master.002.patch, HBASE-20625.master.003.patch,
> HBASE-20625.master.004.patch, HBASE-20625.master.005.patch,
> HBASE-20625.master.006.patch, HBASE-20625.master.007.patch,
> HBASE-20625.master.008.patch
>
>
> Currently I'm working on export HLog to another FileSystem, then I found the
> code of WALCellCodec and its related classes is not that clean. And there
> are several TODOs. Thus I tried to refactor the code based one these TODOs.
> e.g.
> {code}
> // TODO: it sucks that compression context is in WAL.Entry. It'd be nice if
> it was here.
> // Dictionary could be gotten by enum; initially, based on enum,
> context would create
> // an array of dictionaries.
> static class BaosAndCompressor extends ByteArrayOutputStream implements
> ByteStringCompressor {
> public ByteString toByteString() {
> // We need this copy to create the ByteString as the byte[] 'buf' is
> not immutable. We reuse
> // them.
> return ByteString.copyFrom(this.buf, 0, this.count);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20733) QABot should run checkstyle tests if the checkstyle configs change
[
https://issues.apache.org/jira/browse/HBASE-20733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512984#comment-16512984
]
Hadoop QA commented on HBASE-20733:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
2s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} The patch generated 0 new + 0 unchanged - 2 fixed =
0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}158m 50s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
| | hadoop.hbase.mapred.TestTableInputFormat |
| | hadoop.hbase.mapreduce.TestRowCounter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927860/HBASE-20733.0.patch |
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
xml |
| uname | Linux 940ed822ae11 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 0b28155d27 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/13252/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13252/testReport/ |
| Max. process+thread count | 4907 (vs. ulimit of 1) |
| modules | C: hbase-checkstyle . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13252/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> QABot should run checkstyle tests if the checkstyle configs change
> --
>
> Key: HBASE-20733
> URL: https://issues.apache.org/jira/browse/HBASE-20733
> Project: HBase
> Issue Type: Improvement
> Components: build, community
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20733.0.patch
>
>
> right now we only do
[jira] [Commented] (HBASE-20722) Make RegionServerTracker only depend on children changed event
[
https://issues.apache.org/jira/browse/HBASE-20722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512981#comment-16512981
]
Hudson commented on HBASE-20722:
Results for branch master
[build #365 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/365/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/365//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Make RegionServerTracker only depend on children changed event
> --
>
> Key: HBASE-20722
> URL: https://issues.apache.org/jira/browse/HBASE-20722
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20722-v1.patch, HBASE-20722.patch
>
>
> For now we will use children changed event for adding RS, and node deleted
> event for removing RS. Actually, children changed can also be used for
> deleting RS, this will make it easier to control as we do not need to deal
> with the concurrency issue between the children changed and node deleted
> event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20734: Component/s: wal Recovery MTTR > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Priority: Major > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20676) Give .hbase-snapshot proper ownership upon directory creation
[ https://issues.apache.org/jira/browse/HBASE-20676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20676: --- Labels: snapshot (was: ) > Give .hbase-snapshot proper ownership upon directory creation > - > > Key: HBASE-20676 > URL: https://issues.apache.org/jira/browse/HBASE-20676 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Priority: Major > Labels: snapshot > > This is continuation of the discussion over HBASE-20668. > Tthe .hbase-snapshot directory is not created at cluster startup. Normally it > is created when snapshot operation is initiated. > However, if before any snapshot operation is performed, some non-super user > from another cluster conducts ExportSnapshot to this cluster, the > .hbase-snapshot directory would be created as that user. > (This is just one scenario that can lead to wrong ownership) > This JIRA is to seek proper way(s) to ensure that .hbase-snapshot directory > would always carry proper onwership and permission upon creation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512958#comment-16512958
]
Josh Elser commented on HBASE-20723:
Ok, thanks for that explanation, Ted. Leaves me wondering how Stephen and Zach
didn't notice this one before, but I'll leave it to them to share with us :)
> WALSplitter uses the rootDir, which is walDir, as the tableDir root path.
> -
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: hbase
>Affects Versions: 1.1.2
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Major
> Attachments: 20723.v1.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt,
> 20723.v5.txt, 20723.v5.txt, 20723.v6.txt, logs.zip
>
>
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20735) Invalid validation of coprocessor whitelist
[
https://issues.apache.org/jira/browse/HBASE-20735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512942#comment-16512942
]
Mike Drob commented on HBASE-20735:
---
is this an issue on branch-1, branch-2, or both?
> Invalid validation of coprocessor whitelist
> ---
>
> Key: HBASE-20735
> URL: https://issues.apache.org/jira/browse/HBASE-20735
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Jagadeesh Anabathula
>Assignee: Clay B.
>Priority: Major
> Labels: security
>
> Per HBASE-16700, coprocessors can be present only in whitelisted paths.
> It validates for every new coprocessor, if jar's path is in whitelist paths.
> It is currently validating only the first coprocessor that is set to a
> table. All the coprocessors that are added after that are not validated and
> allows path other than that are whitelisted.
> In my case, I have hbase.coprocessor.region.whitelist.paths as
> /tmp/**,*/tmp/coprocessors/*
> Following works fine
> {code}
> hbase(main):001:0> create 'test_coprocessors', 'c'
> 0 row(s) in 1.7540 seconds
> => Hbase::Table - test_coprocessors
> hbase(main):002:0> alter 'test_coprocessors', METHOD => 'table_att',
> 'COPROCESSOR' =>
> 'hdfs:/tmp/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observers.PrefixedDataFilter|100|prefix=P'
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 2.1250 seconds
> hbase(main):003:0> alter 'test_coprocessors', METHOD => 'table_att',
> 'COPROCESSOR' =>
> 'hdfs:/user/hbase/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observers.PrefixedDataFilter|100|prefix=P'
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 1.9690 seconds
> hbase(main):004:0> desc 'test_coprocessors'
> Table test_coprocessors is ENABLED
> test_coprocessors, {TABLE_ATTRIBUTES => {METADATA => {'COPROCESSOR$1' =>
> 'hdfs:/tmp/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observer
> s.PrefixedDataFilter|100|prefix=P', 'COPROCESSOR$2' =>
> 'hdfs:/user/hbase/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observers.Prefi
> xedDataFilter|100|prefix=P'}}
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'c', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true',
> BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
> 1 row(s) in 0.0220 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
