[jira] [Commented] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[
https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514992#comment-16514992
]
Hadoop QA commented on HBASE-20727:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}132m 31s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.TestRegionReplicasAreDistributed |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20727 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928098/HBASE-20727.005.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 22c0d7b2e525 3.13.0-139-generic #188-Ubuntu
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514987#comment-16514987
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #49 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/49/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/49//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/49//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/49//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20743) ASF License warnings for branch-1
[
https://issues.apache.org/jira/browse/HBASE-20743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514986#comment-16514986
]
Sean Busbey commented on HBASE-20743:
-
files in {{target}} should be getting ignored already. does this reproduce
locally?
> ASF License warnings for branch-1
> -
>
> Key: HBASE-20743
> URL: https://issues.apache.org/jira/browse/HBASE-20743
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> From
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350/artifact/output-general/patch-asflicense-problems.txt
> :
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header:
> !? hbase-error-prone/target/checkstyle-result.xml
> !?
> hbase-error-prone/target/classes/META-INF/services/com.google.errorprone.bugpatterns.BugChecker
> !?
> hbase-error-prone/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
> !?
> hbase-error-prone/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
> {code}
> Looks like they should be excluded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514981#comment-16514981
]
Reid Chan commented on HBASE-20732:
---
Quite different between local results..
> Shutdown scan pool when master is stopped.
> --
>
> Key: HBASE-20732
> URL: https://issues.apache.org/jira/browse/HBASE-20732
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-20732.master.001.patch
>
>
> If master is stopped, {{DirScanPool}} is kept open. This is found by
> [~chia7712] when reviewing HBASE-20352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20737) put collection into ArrayList instead of addAll function
[ https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-20737: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Push the patch to master and branch-2. [~taiyinglee] Feel free to reopen this Jira if you have free time to address [~mdrob]'s comment. > put collection into ArrayList instead of addAll function > > > Key: HBASE-20737 > URL: https://issues.apache.org/jira/browse/HBASE-20737 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: taiyinglee >Assignee: taiyinglee >Priority: Trivial > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch > > > [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html] > [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html] > > /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java > change > List> entries = new ArrayList<>(); > entries.addAll(lastSent.entrySet()); > to > List> entries = new > ArrayList<>(lastSent.entrySet()); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514974#comment-16514974
]
Hudson commented on HBASE-20723:
Results for branch branch-1.4
[build #356 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/356/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/356//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/356//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/356//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20679) Add the ability to compile JSP dynamically in Jetty
[
https://issues.apache.org/jira/browse/HBASE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-20679:
---
Attachment: HBASE-20679.004.patch
> Add the ability to compile JSP dynamically in Jetty
> ---
>
> Key: HBASE-20679
> URL: https://issues.apache.org/jira/browse/HBASE-20679
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20679.002.patch, HBASE-20679.003.patch,
> HBASE-20679.004.patch, HBASE-20679.patch
>
>
> As discussed in HBASE-20617, adding the ability to dynamically compile jsp
> enable us to do some hot fix.
> For example, several days ago, in our testing HBase-2.0 cluster,
> procedureWals were corrupted due to some unknown reasons. After restarting
> the cluster, since some procedures(AssignProcedure for example) were
> corrupted and couldn't be replayed. Some regions were stuck in RIT forever.
> We couldn't use HBCK since it haven't support AssignmentV2 yet. As a matter
> of fact, the namespace region was not online, so the master was not inited,
> we even couldn't use shell command like assign/move. But, we wrote a jsp and
> fix this issue easily. The jsp file is like this:
> {code:java}
> <%
> String action = request.getParameter("action");
> HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
> List offlineRegionsToAssign = new ArrayList<>();
> List regionRITs =
> master.getAssignmentManager()
> .getRegionStates().getRegionsInTransition();
> for (RegionStates.RegionStateNode regionStateNode : regionRITs) {
> // if regionStateNode don't have a procedure attached, but meta state
> shows
> // this region is in RIT, that means the previous procedure may be
> corrupted
> // we need to create a new assignProcedure to assign them
> if (!regionStateNode.isInTransition()) {
> offlineRegionsToAssign.add(regionStateNode.getRegionInfo());
> out.println("RIT region:" + regionStateNode);
> }
> }
> // Assign offline regions. Uses round-robin.
> if ("fix".equals(action) && offlineRegionsToAssign.size() > 0) {
>
> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().
> createRoundRobinAssignProcedures(offlineRegionsToAssign));
> } else {
> out.println("use ?action=fix to fix RIT regions");
> }
> %>
> {code}
> Above it is only one example we can do if we have the ability to compile jsp
> dynamically. We think it is very useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20727) Persist FlushedSequenceId to speed up WAL split after cluster restart
[ https://issues.apache.org/jira/browse/HBASE-20727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allan Yang updated HBASE-20727: --- Attachment: HBASE-20727.005.patch > Persist FlushedSequenceId to speed up WAL split after cluster restart > - > > Key: HBASE-20727 > URL: https://issues.apache.org/jira/browse/HBASE-20727 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20727.002.patch, HBASE-20727.003.patch, > HBASE-20727.004.patch, HBASE-20727.005.patch, HBASE-20727.patch > > > We use flushedSequenceIdByRegion and storeFlushedSequenceIdsByRegion in > ServerManager to record the latest flushed seqids of regions and stores. So > during log split, we can use seqids stored in those maps to filter out the > edits which do not need to be replayed. But, those maps are not persisted. > After cluster restart or master restart, info of flushed seqids are all lost. > Here I offer a way to persist those info to HDFS, even if master restart, we > can still use those info to filter WAL edits and then to speed up replay. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20745) Log when master proc wal rolls
[
https://issues.apache.org/jira/browse/HBASE-20745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514932#comment-16514932
]
Hadoop QA commented on HBASE-20745:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
45s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 48s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
41s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m
9s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20745 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928092/HBASE-20745.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 4867bec8e62c 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ac5bb8155b |
| maven | version:
[jira] [Commented] (HBASE-20735) Invalid validation of coprocessor whitelist
[
https://issues.apache.org/jira/browse/HBASE-20735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514910#comment-16514910
]
Clay B. commented on HBASE-20735:
-
This was seen on Hortonworks 2.6.3 as I understand it so roughly 1.1.2 based
and thus not a current release of any type. I am working on a test to reproduce
as I have not seen the cluster in question yet.
> Invalid validation of coprocessor whitelist
> ---
>
> Key: HBASE-20735
> URL: https://issues.apache.org/jira/browse/HBASE-20735
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Reporter: Jagadeesh Anabathula
>Assignee: Clay B.
>Priority: Major
> Labels: security
>
> Per HBASE-16700, coprocessors can be present only in whitelisted paths.
> It validates for every new coprocessor, if jar's path is in whitelist paths.
> It is currently validating only the first coprocessor that is set to a
> table. All the coprocessors that are added after that are not validated and
> allows path other than that are whitelisted.
> In my case, I have hbase.coprocessor.region.whitelist.paths as
> /tmp/**,*/tmp/coprocessors/*
> Following works fine
> {code}
> hbase(main):001:0> create 'test_coprocessors', 'c'
> 0 row(s) in 1.7540 seconds
> => Hbase::Table - test_coprocessors
> hbase(main):002:0> alter 'test_coprocessors', METHOD => 'table_att',
> 'COPROCESSOR' =>
> 'hdfs:/tmp/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observers.PrefixedDataFilter|100|prefix=P'
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 2.1250 seconds
> hbase(main):003:0> alter 'test_coprocessors', METHOD => 'table_att',
> 'COPROCESSOR' =>
> 'hdfs:/user/hbase/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observers.PrefixedDataFilter|100|prefix=P'
> Updating all regions with the new schema...
> 1/1 regions updated.
> Done.
> 0 row(s) in 1.9690 seconds
> hbase(main):004:0> desc 'test_coprocessors'
> Table test_coprocessors is ENABLED
> test_coprocessors, {TABLE_ATTRIBUTES => {METADATA => {'COPROCESSOR$1' =>
> 'hdfs:/tmp/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observer
> s.PrefixedDataFilter|100|prefix=P', 'COPROCESSOR$2' =>
> 'hdfs:/user/hbase/coprocessors/coprocessors-0.4.0.jar|com.test.hbase.coprocessors.observers.Prefi
> xedDataFilter|100|prefix=P'}}
> COLUMN FAMILIES DESCRIPTION
> {NAME => 'c', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false',
> KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL =>
> 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true',
> BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}
> 1 row(s) in 0.0220 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20745) Log when master proc wal rolls
[ https://issues.apache.org/jira/browse/HBASE-20745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20745: -- Fix Version/s: 2.0.2 > Log when master proc wal rolls > -- > > Key: HBASE-20745 > URL: https://issues.apache.org/jira/browse/HBASE-20745 > Project: HBase > Issue Type: Sub-task > Components: debugging >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: HBASE-20745.master.001.patch > > > Emit when we roll master proc WAL so can see when they happen. Want to > correlate instances of corruption w/ events on Master. Currently hard to do > on a server where log-level is INFO (default for many deploys). > Also, we log STUCK regions every 5 seconds. If a bundle of regions get stuck, > we can log so frequently, we roll away where the problem happened so lose the > chance to debug. Let me fix that too > Need both debugging instances of parent issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20745) Log when master proc wal rolls
[ https://issues.apache.org/jira/browse/HBASE-20745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20745: -- Attachment: HBASE-20745.master.001.patch > Log when master proc wal rolls > -- > > Key: HBASE-20745 > URL: https://issues.apache.org/jira/browse/HBASE-20745 > Project: HBase > Issue Type: Sub-task > Components: debugging >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: HBASE-20745.master.001.patch > > > Emit when we roll master proc WAL so can see when they happen. Want to > correlate instances of corruption w/ events on Master. Currently hard to do > on a server where log-level is INFO (default for many deploys). > Also, we log STUCK regions every 5 seconds. If a bundle of regions get stuck, > we can log so frequently, we roll away where the problem happened so lose the > chance to debug. Let me fix that too > Need both debugging instances of parent issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20745) Log when master proc wal rolls
[ https://issues.apache.org/jira/browse/HBASE-20745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20745: -- Status: Patch Available (was: Open) > Log when master proc wal rolls > -- > > Key: HBASE-20745 > URL: https://issues.apache.org/jira/browse/HBASE-20745 > Project: HBase > Issue Type: Sub-task > Components: debugging >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.2 > > Attachments: HBASE-20745.master.001.patch > > > Emit when we roll master proc WAL so can see when they happen. Want to > correlate instances of corruption w/ events on Master. Currently hard to do > on a server where log-level is INFO (default for many deploys). > Also, we log STUCK regions every 5 seconds. If a bundle of regions get stuck, > we can log so frequently, we roll away where the problem happened so lose the > chance to debug. Let me fix that too > Need both debugging instances of parent issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20745) Log when master proc wal rolls
stack created HBASE-20745: - Summary: Log when master proc wal rolls Key: HBASE-20745 URL: https://issues.apache.org/jira/browse/HBASE-20745 Project: HBase Issue Type: Sub-task Components: debugging Reporter: stack Assignee: stack Emit when we roll master proc WAL so can see when they happen. Want to correlate instances of corruption w/ events on Master. Currently hard to do on a server where log-level is INFO (default for many deploys). Also, we log STUCK regions every 5 seconds. If a bundle of regions get stuck, we can log so frequently, we roll away where the problem happened so lose the chance to debug. Let me fix that too Need both debugging instances of parent issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514894#comment-16514894
]
Hudson commented on HBASE-20723:
Results for branch master
[build #367 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/367/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20737) put collection into ArrayList instead of addAll function
[
https://issues.apache.org/jira/browse/HBASE-20737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514892#comment-16514892
]
Hudson commented on HBASE-20737:
Results for branch master
[build #367 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/367/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> put collection into ArrayList instead of addAll function
>
>
> Key: HBASE-20737
> URL: https://issues.apache.org/jira/browse/HBASE-20737
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: taiyinglee
>Assignee: taiyinglee
>Priority: Trivial
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20737.v0.patch, HBASE-20737.v0.patch
>
>
> [https://docs.oracle.com/javase/7/docs/api/java/util/Collection.html]
> [https://docs.oracle.com/javase/7/docs/api/java/util/ArrayList.html]
> [https://docs.oracle.com/javase/7/docs/api/java/util/Set.html]
>
> /hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ClusterStatusPublisher.java
> change
> List> entries = new ArrayList<>();
> entries.addAll(lastSent.entrySet());
> to
> List> entries = new
> ArrayList<>(lastSent.entrySet());
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20730) Add pv2 and amv2 chapters to refguide
[
https://issues.apache.org/jira/browse/HBASE-20730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514893#comment-16514893
]
Hudson commented on HBASE-20730:
Results for branch master
[build #367 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/367/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add pv2 and amv2 chapters to refguide
> -
>
> Key: HBASE-20730
> URL: https://issues.apache.org/jira/browse/HBASE-20730
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20730.master.001.patch,
> HBASE-20730.master.002.patch, HBASE-20730.master.003.patch,
> HBASE-20730.master.004.patch, HBASE-20730.master.005.patch
>
>
> Found some doc I'd made on pv2 and amv2. Reads better than nothing. Made for
> dev'y audience. No harm having chapters in the refguide, our bucket of
> all-things doc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20681) IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
[
https://issues.apache.org/jira/browse/HBASE-20681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514891#comment-16514891
]
Hudson commented on HBASE-20681:
Results for branch master
[build #367 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/367/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/367//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core
> ---
>
> Key: HBASE-20681
> URL: https://issues.apache.org/jira/browse/HBASE-20681
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Reporter: Romil Choksi
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: HBASE-20681.001.patch, HBASE-20681.002.patch,
> HBASE-20681.003.patch, HBASE-20681.004.patch, HBASE-20681.004.patch
>
>
> HADOOP-15406 marked mockito and junit as test-only dependencies which, I
> believe, has stopped them from being included in a stock Hadoop classpath.
> Prior, you'd get hamcrest at {{share/hadoop/common/lib/hamcrest-core-1.3.jar}}
> However, we depend on it being there for our junit in hbase-it:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.1:tree (default-cli) @ hbase-it ---
> [INFO] org.apache.hbase:hbase-it:jar:2.0.1-SNAPSHOT
> [INFO] +- junit:junit:jar:4.12:test
> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
> {noformat}
> We need to make sure we include it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514875#comment-16514875
]
Hadoop QA commented on HBASE-20732:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
1s{color} | {color:red} hbase-server: The patch generated 2 new + 173 unchanged
- 0 fixed = 175 total (was 173) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 52s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 25s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.cleaner.TestCleanerChore |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20732 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12928085/HBASE-20732.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6668d0015f6f 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11
12:16:42 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ac5bb8155b |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13284/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/13284/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Created] (HBASE-20744) Address FindBugs warnings in branch-1
Ted Yu created HBASE-20744:
--
Summary: Address FindBugs warnings in branch-1
Key: HBASE-20744
URL: https://issues.apache.org/jira/browse/HBASE-20744
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
>From
>https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350//JDK8_Nightly_Build_Report_(Hadoop2)/
> :
{code}
FindBugsmodule:hbase-common
Inconsistent synchronization of
org.apache.hadoop.hbase.io.encoding.EncodedDataBlock$BufferGrabbingByteArrayOutputStream.ourBytes;
locked 50% of time Unsynchronized access at EncodedDataBlock.java:50% of time
Unsynchronized access at EncodedDataBlock.java:[line 258]
{code}
{code}
FindBugsmodule:hbase-hadoop2-compat
java.util.concurrent.ScheduledThreadPoolExecutor stored into non-transient
field MetricsExecutorImpl$ExecutorSingleton.scheduler At
MetricsExecutorImpl.java:MetricsExecutorImpl$ExecutorSingleton.scheduler At
MetricsExecutorImpl.java:[line 51]
{code}
{code}
FindBugsmodule:hbase-server
instanceof will always return false in
org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(Region, int,
int, int), since a org.apache.hadoop.hbase.quotas.RpcThrottlingException can't
be a org.apache.hadoop.hbase.quotas.ThrottlingException At
RegionServerQuotaManager.java:in
org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(Region, int,
int, int), since a org.apache.hadoop.hbase.quotas.RpcThrottlingException can't
be a org.apache.hadoop.hbase.quotas.ThrottlingException At
RegionServerQuotaManager.java:[line 193]
instanceof will always return true for all non-null values in
org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(Region, int,
int, int), since all org.apache.hadoop.hbase.quotas.RpcThrottlingException are
instances of org.apache.hadoop.hbase.quotas.RpcThrottlingException At
RegionServerQuotaManager.java:for all non-null values in
org.apache.hadoop.hbase.quotas.RegionServerQuotaManager.checkQuota(Region, int,
int, int), since all org.apache.hadoop.hbase.quotas.RpcThrottlingException are
instances of org.apache.hadoop.hbase.quotas.RpcThrottlingException At
RegionServerQuotaManager.java:[line 199]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20743) ASF License warnings for branch-1
Ted Yu created HBASE-20743:
--
Summary: ASF License warnings for branch-1
Key: HBASE-20743
URL: https://issues.apache.org/jira/browse/HBASE-20743
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
>From
>https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350/artifact/output-general/patch-asflicense-problems.txt
> :
{code}
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header:
!? hbase-error-prone/target/checkstyle-result.xml
!?
hbase-error-prone/target/classes/META-INF/services/com.google.errorprone.bugpatterns.BugChecker
!?
hbase-error-prone/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
!?
hbase-error-prone/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
{code}
Looks like they should be excluded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20732:
--
Status: Patch Available (was: Open)
> Shutdown scan pool when master is stopped.
> --
>
> Key: HBASE-20732
> URL: https://issues.apache.org/jira/browse/HBASE-20732
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-20732.master.001.patch
>
>
> If master is stopped, {{DirScanPool}} is kept open. This is found by
> [~chia7712] when reviewing HBASE-20352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan updated HBASE-20732:
--
Attachment: HBASE-20732.master.001.patch
> Shutdown scan pool when master is stopped.
> --
>
> Key: HBASE-20732
> URL: https://issues.apache.org/jira/browse/HBASE-20732
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-20732.master.001.patch
>
>
> If master is stopped, {{DirScanPool}} is kept open. This is found by
> [~chia7712] when reviewing HBASE-20352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514856#comment-16514856
]
Reid Chan commented on HBASE-20732:
---
Let's see what QA say.
> Shutdown scan pool when master is stopped.
> --
>
> Key: HBASE-20732
> URL: https://issues.apache.org/jira/browse/HBASE-20732
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Attachments: HBASE-20732.master.001.patch
>
>
> If master is stopped, {{DirScanPool}} is kept open. This is found by
> [~chia7712] when reviewing HBASE-20352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514854#comment-16514854
]
Hudson commented on HBASE-20723:
Results for branch branch-1
[build #350 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/350//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514839#comment-16514839
]
Sean Busbey commented on HBASE-20332:
-
ugh. checkstyle in hte precommit run broke with a complaint that the
suppression isn't valid. but it works locally? trying to figure out the
difference.
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch,
> HBASE-20332.5.patch, HBASE-20332.6.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20679) Add the ability to compile JSP dynamically in Jetty
[
https://issues.apache.org/jira/browse/HBASE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514815#comment-16514815
]
Allan Yang commented on HBASE-20679:
[~mdrob], glad that someone is still interested in this patch. I thought this
patch won't be checked in due to safety issues. I will fix the precommit issues
in a few days anyway. Thanks for reminding me.
> Add the ability to compile JSP dynamically in Jetty
> ---
>
> Key: HBASE-20679
> URL: https://issues.apache.org/jira/browse/HBASE-20679
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20679.002.patch, HBASE-20679.003.patch,
> HBASE-20679.patch
>
>
> As discussed in HBASE-20617, adding the ability to dynamically compile jsp
> enable us to do some hot fix.
> For example, several days ago, in our testing HBase-2.0 cluster,
> procedureWals were corrupted due to some unknown reasons. After restarting
> the cluster, since some procedures(AssignProcedure for example) were
> corrupted and couldn't be replayed. Some regions were stuck in RIT forever.
> We couldn't use HBCK since it haven't support AssignmentV2 yet. As a matter
> of fact, the namespace region was not online, so the master was not inited,
> we even couldn't use shell command like assign/move. But, we wrote a jsp and
> fix this issue easily. The jsp file is like this:
> {code:java}
> <%
> String action = request.getParameter("action");
> HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
> List offlineRegionsToAssign = new ArrayList<>();
> List regionRITs =
> master.getAssignmentManager()
> .getRegionStates().getRegionsInTransition();
> for (RegionStates.RegionStateNode regionStateNode : regionRITs) {
> // if regionStateNode don't have a procedure attached, but meta state
> shows
> // this region is in RIT, that means the previous procedure may be
> corrupted
> // we need to create a new assignProcedure to assign them
> if (!regionStateNode.isInTransition()) {
> offlineRegionsToAssign.add(regionStateNode.getRegionInfo());
> out.println("RIT region:" + regionStateNode);
> }
> }
> // Assign offline regions. Uses round-robin.
> if ("fix".equals(action) && offlineRegionsToAssign.size() > 0) {
>
> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().
> createRoundRobinAssignProcedures(offlineRegionsToAssign));
> } else {
> out.println("use ?action=fix to fix RIT regions");
> }
> %>
> {code}
> Above it is only one example we can do if we have the ability to compile jsp
> dynamically. We think it is very useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19064) Synchronous replication for HBase
[
https://issues.apache.org/jira/browse/HBASE-19064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514781#comment-16514781
]
Hudson commented on HBASE-19064:
Results for branch HBASE-19064
[build #163 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/163/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/163//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/163//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-19064/163//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Synchronous replication for HBase
> -
>
> Key: HBASE-19064
> URL: https://issues.apache.org/jira/browse/HBASE-19064
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> The guys from Alibaba made a presentation on HBaseCon Asia about the
> synchronous replication for HBase. We(Xiaomi) think this is a very useful
> feature for HBase so we want to bring it into the community version.
> This is a big feature so we plan to do it in a feature branch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514763#comment-16514763 ] Anoop Sam John commented on HBASE-20716: Ya we can clean it up and make like what we have in Bytes (2 impls based on Unsafe or PureJava). > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514757#comment-16514757
]
Hudson commented on HBASE-20723:
Results for branch branch-2
[build #868 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/868/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/868//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/868//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/868//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
