[jira] [Commented] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[
https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671112#comment-16671112
]
Hudson commented on HBASE-21191:
Results for branch branch-2.0
[build #1045 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1045/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1045//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1045//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1045//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add a holding-pattern if no assign for meta or namespace (Can happen if
> masterprocwals have been cleared).
> --
>
> Key: HBASE-21191
> URL: https://issues.apache.org/jira/browse/HBASE-21191
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1, 2.0.3
>
> Attachments: HBASE-21191.branch-2.0.001.patch,
> HBASE-21191.branch-2.1.001.patch, HBASE-21191.branch-2.1.002.patch,
> HBASE-21191.branch-2.1.003.patch, HBASE-21191.branch-2.1.004.patch,
> HBASE-21191.branch-2.1.005.patch, HBASE-21191.branch-2.1.006.patch,
> HBASE-21191.branch-2.1.007.patch
>
>
> If the masterprocwals have been removed -- operator error, hdfs dataloss, or
> because we have gotten ourselves into a pathological state where we have
> hundreds of masterprocwals too process and it is taking too long so we just
> want to startover -- then master startup will have a dilemma. Master startup
> needs hbase:meta to be online. If the masterprocwals have been removed, there
> may be no outstanding assign or a servercrashprocedure with coverage for
> hbase:meta (I ran into this issue repeatedly in internal testing purging
> masterprocwals on a large test cluster). Worse, when master startup cannot
> find an online hbase:meta, it exits after exhausting the RPC retries.
> So, we need a holding-pattern for master startup if hbase:meta is not online
> if only so an operator can schedule an assign for meta or so they can assign
> fixup procedures (HBASE-21035 has discussion on why we cannot just
> auto-schedule an assign of meta).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671101#comment-16671101 ] stack commented on HBASE-21035: --- This has been haunting me. [~elserj] had an issue he hoisted up on to the dev list where namespace would not deploy. Then there is the issue over in the heatmaps JIRA. There is a messy internal issue where we are trying to get more details that hints that it could be a prob. in update. Let me try it in morning. > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[ https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671094#comment-16671094 ] stack commented on HBASE-21388: --- +1 on patch. +1 for branch-2.1 and branch-2.0. Thanks for ping [~zghaobac] > No need to instantiate MemStoreLAB for master which not carry table > --- > > Key: HBASE-21388 > URL: https://issues.apache.org/jira/browse/HBASE-21388 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21388.master.001.patch, > HBASE-21388.master.002.patch, HBASE-21388.master.003.patch, > HBASE-21388.master.004.patch, HBASE-21388.master.004.patch > > > We found this log in our master. > 2018-10-26,10:00:00,449 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating data > MemStoreChunkPool with chunk size 2 MB, max count 737, initial count 0 > 2018-10-26,10:00:00,452 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating index > MemStoreChunkPool with chunk size 204.80 KB, max count 819, initial count 0 > > Same with HBASE-21290, we don't need to instantiate MemStore for master which > not carry table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671092#comment-16671092
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #35 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/35/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/35//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/35//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/35//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21417) Pre commit build is broken due to surefire plugin crashes
[
https://issues.apache.org/jira/browse/HBASE-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671091#comment-16671091
]
stack commented on HBASE-21417:
---
+1 Try it.
> Pre commit build is broken due to surefire plugin crashes
> -
>
> Key: HBASE-21417
> URL: https://issues.apache.org/jira/browse/HBASE-21417
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments: HBASE-21417.patch
>
>
> The recent builds are all failed with
> {noformat}
> [ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-rsgroup &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions
> -Dhbase.build.id=2018-10-31T11:09:36Z -Xmx2800m
> -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -jar
> /testptch/hbase/hbase-rsgroup/target/surefire/surefirebooter3799876849632796400.jar
> /testptch/hbase/hbase-rsgroup/target/surefire
> 2018-10-31T11-09-52_393-jvmRun1 surefire4495583426680149115tmp
> surefire_05657090267882138674tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 1
> {noformat}
> [~risdenk] provided some useful reference:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925
> https://github.com/carlossg/docker-maven/issues/90
> https://github.com/carlossg/docker-maven/issues/92
> It seems to be an OpenJDK issue.
> Let's see if there are any workarounds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian reassigned HBASE-21413: Assignee: Allan Yang (was: Jingyun Tian) > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Allan Yang >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21035) Meta Table should be able to online even if all procedures are lost
[ https://issues.apache.org/jira/browse/HBASE-21035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671049#comment-16671049 ] Allan Yang commented on HBASE-21035: The problem mentioned in this issue need to be discussed again. I tried to update a 1.x cluster to 2.x. Firstly, I stopped all servers(may leave some splitting wal dirs). Then I started 2.x in place, but since now when master starting, we totally count on the procedures, and won't care about the splitting WAL dirs, some data may loss. And there is no SCPs to bring meta online. This would be a problem for people who want to upgrade their clusters from 1.x to 2.x. [~stack],[~Apache9] > Meta Table should be able to online even if all procedures are lost > --- > > Key: HBASE-21035 > URL: https://issues.apache.org/jira/browse/HBASE-21035 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Attachments: HBASE-21035.branch-2.0.001.patch, > HBASE-21035.branch-2.1.001.patch > > > After HBASE-20708, we changed the way we init after master starts. It will > only check WAL dirs and compare to Zookeeper RS nodes to decide which server > need to expire. For servers which's dir is ending with 'SPLITTING', we assure > that there will be a SCP for it. > But, if the server with the meta region crashed before master restarts, and > if all the procedure wals are lost (due to bug, or deleted manually, > whatever), the new restarted master will be stuck when initing. Since no one > will bring meta region online. > Although it is an anomaly case, but I think no matter what happens, we need > to online meta region. Otherwise, we are sitting ducks, noting can be done. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21301) Heatmap for key access patterns
[ https://issues.apache.org/jira/browse/HBASE-21301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671045#comment-16671045 ] Allan Yang edited comment on HBASE-21301 at 11/1/18 2:49 AM: - [~archana.katiyar] do you start a brand new cluster( no zk node) or start from an existing one? was (Author: allan163): [~archana.katiyar] do you start a brand new cluster( no zk node) or start from a existing one? > Heatmap for key access patterns > --- > > Key: HBASE-21301 > URL: https://issues.apache.org/jira/browse/HBASE-21301 > Project: HBase > Issue Type: Improvement >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21301.v1.patch > > > Google recently released a beta feature for Cloud Bigtable which presents a > heat map of the keyspace. *Given how hotspotting comes up now and again here, > this is a good idea for giving HBase ops a tool to be proactive about it.* > >>> > Additionally, we are announcing the beta version of Key Visualizer, a > visualization tool for Cloud Bigtable key access patterns. Key Visualizer > helps debug performance issues due to unbalanced access patterns across the > key space, or single rows that are too large or receiving too much read or > write activity. With Key Visualizer, you get a heat map visualization of > access patterns over time, along with the ability to zoom into specific key > or time ranges, or select a specific row to find the full row key ID that's > responsible for a hotspot. Key Visualizer is automatically enabled for Cloud > Bigtable clusters with sufficient data or activity, and does not affect Cloud > Bigtable cluster performance. > <<< > From > [https://cloudplatform.googleblog.com/2018/07/on-gcp-your-database-your-way.html] > (Copied this description from the write-up by [~apurtell], thanks Andrew.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21301) Heatmap for key access patterns
[ https://issues.apache.org/jira/browse/HBASE-21301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671045#comment-16671045 ] Allan Yang commented on HBASE-21301: [~archana.katiyar] do you start a brand new cluster( no zk node) or start from a existing one? > Heatmap for key access patterns > --- > > Key: HBASE-21301 > URL: https://issues.apache.org/jira/browse/HBASE-21301 > Project: HBase > Issue Type: Improvement >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21301.v1.patch > > > Google recently released a beta feature for Cloud Bigtable which presents a > heat map of the keyspace. *Given how hotspotting comes up now and again here, > this is a good idea for giving HBase ops a tool to be proactive about it.* > >>> > Additionally, we are announcing the beta version of Key Visualizer, a > visualization tool for Cloud Bigtable key access patterns. Key Visualizer > helps debug performance issues due to unbalanced access patterns across the > key space, or single rows that are too large or receiving too much read or > write activity. With Key Visualizer, you get a heat map visualization of > access patterns over time, along with the ability to zoom into specific key > or time ranges, or select a specific row to find the full row key ID that's > responsible for a hotspot. Key Visualizer is automatically enabled for Cloud > Bigtable clusters with sufficient data or activity, and does not affect Cloud > Bigtable cluster performance. > <<< > From > [https://cloudplatform.googleblog.com/2018/07/on-gcp-your-database-your-way.html] > (Copied this description from the write-up by [~apurtell], thanks Andrew.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21417) Pre commit build is broken due to surefire plugin crashes
[
https://issues.apache.org/jira/browse/HBASE-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21417:
--
Component/s: build
> Pre commit build is broken due to surefire plugin crashes
> -
>
> Key: HBASE-21417
> URL: https://issues.apache.org/jira/browse/HBASE-21417
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments: HBASE-21417.patch
>
>
> The recent builds are all failed with
> {noformat}
> [ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-rsgroup &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions
> -Dhbase.build.id=2018-10-31T11:09:36Z -Xmx2800m
> -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -jar
> /testptch/hbase/hbase-rsgroup/target/surefire/surefirebooter3799876849632796400.jar
> /testptch/hbase/hbase-rsgroup/target/surefire
> 2018-10-31T11-09-52_393-jvmRun1 surefire4495583426680149115tmp
> surefire_05657090267882138674tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 1
> {noformat}
> [~risdenk] provided some useful reference:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925
> https://github.com/carlossg/docker-maven/issues/90
> https://github.com/carlossg/docker-maven/issues/92
> It seems to be an OpenJDK issue.
> Let's see if there are any workarounds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21417) Pre commit build is broken due to surefire plugin crashes
[
https://issues.apache.org/jira/browse/HBASE-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21417:
--
Assignee: Duo Zhang
Status: Patch Available (was: Open)
> Pre commit build is broken due to surefire plugin crashes
> -
>
> Key: HBASE-21417
> URL: https://issues.apache.org/jira/browse/HBASE-21417
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Attachments: HBASE-21417.patch
>
>
> The recent builds are all failed with
> {noformat}
> [ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-rsgroup &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions
> -Dhbase.build.id=2018-10-31T11:09:36Z -Xmx2800m
> -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -jar
> /testptch/hbase/hbase-rsgroup/target/surefire/surefirebooter3799876849632796400.jar
> /testptch/hbase/hbase-rsgroup/target/surefire
> 2018-10-31T11-09-52_393-jvmRun1 surefire4495583426680149115tmp
> surefire_05657090267882138674tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 1
> {noformat}
> [~risdenk] provided some useful reference:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925
> https://github.com/carlossg/docker-maven/issues/90
> https://github.com/carlossg/docker-maven/issues/92
> It seems to be an OpenJDK issue.
> Let's see if there are any workarounds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21417) Pre commit build is broken due to surefire plugin crashes
[
https://issues.apache.org/jira/browse/HBASE-21417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21417:
--
Attachment: HBASE-21417.patch
> Pre commit build is broken due to surefire plugin crashes
> -
>
> Key: HBASE-21417
> URL: https://issues.apache.org/jira/browse/HBASE-21417
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Critical
> Attachments: HBASE-21417.patch
>
>
> The recent builds are all failed with
> {noformat}
> [ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-rsgroup &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions
> -Dhbase.build.id=2018-10-31T11:09:36Z -Xmx2800m
> -Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
> -Djava.awt.headless=true -jar
> /testptch/hbase/hbase-rsgroup/target/surefire/surefirebooter3799876849632796400.jar
> /testptch/hbase/hbase-rsgroup/target/surefire
> 2018-10-31T11-09-52_393-jvmRun1 surefire4495583426680149115tmp
> surefire_05657090267882138674tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 1
> {noformat}
> [~risdenk] provided some useful reference:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925
> https://github.com/carlossg/docker-maven/issues/90
> https://github.com/carlossg/docker-maven/issues/92
> It seems to be an OpenJDK issue.
> Let's see if there are any workarounds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[
https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671043#comment-16671043
]
Hadoop QA commented on HBASE-21388:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
0s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 43s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21388 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946474/HBASE-21388.master.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 7a16c8fa5a26 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6d709c0311 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14915/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14915/testReport/ |
| Max. process+thread count | 99 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14915/console |
| Powered by | Apache
[jira] [Created] (HBASE-21417) Pre commit build is broken due to surefire plugin crashes
Duo Zhang created HBASE-21417:
-
Summary: Pre commit build is broken due to surefire plugin crashes
Key: HBASE-21417
URL: https://issues.apache.org/jira/browse/HBASE-21417
Project: HBase
Issue Type: Bug
Reporter: Duo Zhang
The recent builds are all failed with
{noformat}
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /testptch/hbase/hbase-rsgroup &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -enableassertions
-Dhbase.build.id=2018-10-31T11:09:36Z -Xmx2800m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true -jar
/testptch/hbase/hbase-rsgroup/target/surefire/surefirebooter3799876849632796400.jar
/testptch/hbase/hbase-rsgroup/target/surefire 2018-10-31T11-09-52_393-jvmRun1
surefire4495583426680149115tmp surefire_05657090267882138674tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
{noformat}
[~risdenk] provided some useful reference:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925
https://github.com/carlossg/docker-maven/issues/90
https://github.com/carlossg/docker-maven/issues/92
It seems to be an OpenJDK issue.
Let's see if there are any workarounds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21387) Race condition in snapshot cache refreshing leads to loss of snapshot files
[
https://issues.apache.org/jira/browse/HBASE-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671033#comment-16671033
]
Ted Yu commented on HBASE-21387:
In the old code, to simulate the race condition, we can use CountDownLatch.
Here is a sketch:
{code}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/
index 358b4ea..2941400 100644
---
a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
+++
b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
@@ -27,6 +27,7 @@ import java.util.Map;
import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
+import java.util.concurrent.CountDownLatch;
import java.util.concurrent.locks.ReentrantLock;
import org.apache.hadoop.conf.Configuration;
@@ -92,6 +93,8 @@ public class SnapshotFileCache implements Stoppable {
private final SnapshotFileInspector fileInspector;
private final Path snapshotDir;
private final Set cache = new HashSet<>();
+ private final CountDownLatch latchRefresh = new CountDownLatch(1);
+ private final CountDownLatch latchContains = new CountDownLatch(1);
/**
* This is a helper map of information about the snapshot directories so we
don't need to rescan
* them if they haven't changed since the last time we looked.
@@ -180,16 +183,18 @@ public class SnapshotFileCache implements Stoppable {
// cache, but that seems overkill at the moment and isn't necessarily a
bottleneck.
public synchronized Iterable
getUnreferencedFiles(Iterable files,
final SnapshotManager snapshotManager)
- throws IOException {
+ throws IOException, InterruptedException {
List unReferencedFiles = Lists.newArrayList();
List snapshotsInProgress = null;
boolean refreshed = false;
for (FileStatus file : files) {
String fileName = file.getPath().getName();
if (!refreshed && !cache.contains(fileName)) {
+latchRefresh.await();
refreshCache();
refreshed = true;
}
+ latchContains.await();
if (cache.contains(fileName)) {
continue;
}
@@ -226,9 +231,11 @@ public class SnapshotFileCache implements Stoppable {
// 1. update the modified time
this.lastModifiedTime = dirStatus.getModificationTime();
+latchRefresh.countDown();
// 2.clear the cache
this.cache.clear();
+latchContains.countDown();
Map known = new HashMap<>();
// 3. check each of the snapshot directories
{code}
With the fix, the race condition is gone.
> Race condition in snapshot cache refreshing leads to loss of snapshot files
> ---
>
> Key: HBASE-21387
> URL: https://issues.apache.org/jira/browse/HBASE-21387
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Labels: snapshot
> Attachments: 21387.v1.txt
>
>
> During recent report from customer where ExportSnapshot failed:
> {code}
> 2018-10-09 18:54:32,559 ERROR [VerifySnapshot-pool1-t2]
> snapshot.SnapshotReferenceUtil: Can't find hfile:
> 44f6c3c646e84de6a63fe30da4fcb3aa in the real
> (hdfs://in.com:8020/apps/hbase/data/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> or archive
> (hdfs://in.com:8020/apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> directory for the primary table.
> {code}
> We found the following in log:
> {code}
> 2018-10-09 18:54:23,675 DEBUG
> [00:16000.activeMasterManager-HFileCleaner.large-1539035367427]
> cleaner.HFileCleaner: Removing:
> hdfs:///apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa
> from archive
> {code}
> The root cause is race condition surrounding SnapshotFileCache#refreshCache().
> There are two callers of refreshCache: one from RefreshCacheTask#run and the
> other from SnapshotHFileCleaner.
> Let's look at the code of refreshCache:
> {code}
> // if the snapshot directory wasn't modified since we last check, we are
> done
> if (dirStatus.getModificationTime() <= this.lastModifiedTime) return;
> // 1. update the modified time
> this.lastModifiedTime = dirStatus.getModificationTime();
> // 2.clear the cache
> this.cache.clear();
> {code}
> Suppose the RefreshCacheTask runs past the if check and sets
> this.lastModifiedTime
> The cleaner executes refreshCache and returns immediately since
> this.lastModifiedTime matches the modification time of the directory.
> Now RefreshCacheTask clears the cache. By the time the cleaner performs cache
> lookup, the cache is empty.
> Therefore cleaner puts the file into unReferencedFiles - leading to data loss.
--
This message was sent by
[jira] [Commented] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671012#comment-16671012 ] Allan Yang commented on HBASE-21413: [~tianjingyun], OK, I can fix it, will upload a patch later here. > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671011#comment-16671011 ] Jingyun Tian commented on HBASE-21413: -- [~allan163] yes, I think they are the same problem. Would you fix it? Then I will close this issue. Or I can try to fix this, too. > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671006#comment-16671006
]
Hadoop QA commented on HBASE-21415:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
54s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 41s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21415 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946468/HBASE-21415.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2b7677db6e81 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6d709c0311 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14914/artifact/patchprocess/patch-unit-hbase-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14914/testReport/ |
| Max. process+thread count | 97 (vs. ulimit of 1) |
| modules | C: hbase-client U:
[jira] [Commented] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[ https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671004#comment-16671004 ] Guanghao Zhang commented on HBASE-21388: Retry 004 patch for Hadoop QA. And ping [~stack] for branch-2.1 and branch-2.0. Thanks. > No need to instantiate MemStoreLAB for master which not carry table > --- > > Key: HBASE-21388 > URL: https://issues.apache.org/jira/browse/HBASE-21388 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21388.master.001.patch, > HBASE-21388.master.002.patch, HBASE-21388.master.003.patch, > HBASE-21388.master.004.patch, HBASE-21388.master.004.patch > > > We found this log in our master. > 2018-10-26,10:00:00,449 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating data > MemStoreChunkPool with chunk size 2 MB, max count 737, initial count 0 > 2018-10-26,10:00:00,452 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating index > MemStoreChunkPool with chunk size 204.80 KB, max count 819, initial count 0 > > Same with HBASE-21290, we don't need to instantiate MemStore for master which > not carry table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[ https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21388: --- Attachment: HBASE-21388.master.004.patch > No need to instantiate MemStoreLAB for master which not carry table > --- > > Key: HBASE-21388 > URL: https://issues.apache.org/jira/browse/HBASE-21388 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21388.master.001.patch, > HBASE-21388.master.002.patch, HBASE-21388.master.003.patch, > HBASE-21388.master.004.patch, HBASE-21388.master.004.patch > > > We found this log in our master. > 2018-10-26,10:00:00,449 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating data > MemStoreChunkPool with chunk size 2 MB, max count 737, initial count 0 > 2018-10-26,10:00:00,452 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating index > MemStoreChunkPool with chunk size 204.80 KB, max count 819, initial count 0 > > Same with HBASE-21290, we don't need to instantiate MemStore for master which > not carry table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[
https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670979#comment-16670979
]
Hudson commented on HBASE-21322:
Results for branch branch-2.1
[build #564 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/564/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/564//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/564//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/564//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add a scheduleServerCrashProcedure() API to HbckService
> ---
>
> Key: HBASE-21322
> URL: https://issues.apache.org/jira/browse/HBASE-21322
> Project: HBase
> Issue Type: Sub-task
> Components: hbck2
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2
>
> Attachments: HBASE-21322.branch-2.1.001.patch,
> HBASE-21322.master.001.patch, HBASE-21322.master.002.patch,
> HBASE-21322.master.003.patch, HBASE-21322.master.004.patch,
> HBASE-21322.master.005.patch, HBASE-21322.master.006.patch, Screenshot from
> 2018-10-17 13-35-58.png, Screenshot from 2018-10-17 13-38-41.png, Screenshot
> from 2018-10-17 13-47-06.png
>
>
> According to my test, if one RS is down, then all procedure logs are deleted,
> it will lead to that no ServerCrashProcedure is scheduled. And restarting
> master cannot help. Thus we need to schedule a ServerCrashProcedure manually
> to solve the problem. I plan to add a scheduleServerCrashProcedure() API to
> HbckService, then add this API to HBCK2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Laflamme updated HBASE-21415:
--
Attachment: HBASE-21415.patch
Status: Patch Available (was: Open)
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch, HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Laflamme updated HBASE-21415:
--
Status: Open (was: Patch Available)
Canceling patch to update with a new one. Not sure if this is how I should do
this though...
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670929#comment-16670929
]
Philippe Laflamme commented on HBASE-21415:
---
[~xucang] I've updated the patch with a comment about concurrency. Let me know
if it makes sense.
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch, HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[
https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670921#comment-16670921
]
Hudson commented on HBASE-21322:
Results for branch branch-2.0
[build #1043 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1043/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1043//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1043//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1043//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add a scheduleServerCrashProcedure() API to HbckService
> ---
>
> Key: HBASE-21322
> URL: https://issues.apache.org/jira/browse/HBASE-21322
> Project: HBase
> Issue Type: Sub-task
> Components: hbck2
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2
>
> Attachments: HBASE-21322.branch-2.1.001.patch,
> HBASE-21322.master.001.patch, HBASE-21322.master.002.patch,
> HBASE-21322.master.003.patch, HBASE-21322.master.004.patch,
> HBASE-21322.master.005.patch, HBASE-21322.master.006.patch, Screenshot from
> 2018-10-17 13-35-58.png, Screenshot from 2018-10-17 13-38-41.png, Screenshot
> from 2018-10-17 13-47-06.png
>
>
> According to my test, if one RS is down, then all procedure logs are deleted,
> it will lead to that no ServerCrashProcedure is scheduled. And restarting
> master cannot help. Thus we need to schedule a ServerCrashProcedure manually
> to solve the problem. I plan to add a scheduleServerCrashProcedure() API to
> HbckService, then add this API to HBCK2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liubangchen reassigned HBASE-21381:
---
Assignee: liubangchen
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670895#comment-16670895
]
Hadoop QA commented on HBASE-21415:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
4s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21415 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946457/HBASE-21415.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 17def5f132bf 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6d709c0311 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14913/artifact/patchprocess/patch-unit-hbase-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14913/testReport/ |
| Max. process+thread count | 99 (vs. ulimit of 1) |
| modules | C: hbase-client U:
[jira] [Commented] (HBASE-21396) Create 2.1.1 release
[
https://issues.apache.org/jira/browse/HBASE-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670868#comment-16670868
]
stack commented on HBASE-21396:
---
* Vote passed. Result is here:
http://mail-archives.apache.org/mod_mbox/hbase-dev/201810.mbox/%3CCADcMMgGPnpyX_WVEdvx4826LFP%2Ba_qKoN3v8Eb7NbUHwCT%3D_cw%40mail.gmail.com%3E
* Copy from dev to release under apache dist with svn
{code}$ svn info
Path: .
Working Copy Root Path: /Users/stack/checkouts/apache.dist
URL: https://dist.apache.org/repos/dist
Relative URL: ^/
Repository Root: https://dist.apache.org/repos/dist
Repository UUID: 0d268c88-bc11-4956-87df-91683dc98e59
Revision: 30445
Node Kind: directory
Schedule: normal
{code}
* Release up in mvn repo.
* Release in JIRA.
* Update Whimsey: https://reporter.apache.org/addrelease.html?hbase
* Edit src/site/xdoc/downloads.html to add in the new release.
* Wait till website build (or force it).
TODO: Send out notice.
> Create 2.1.1 release
>
>
> Key: HBASE-21396
> URL: https://issues.apache.org/jira/browse/HBASE-21396
> Project: HBase
> Issue Type: Task
> Components: rm
>Reporter: stack
>Assignee: stack
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21374) Backport HBASE-21342 to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670866#comment-16670866
]
Hadoop QA commented on HBASE-21374:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
26s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}147m 50s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterBalanceThrottling |
| | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
| | hadoop.hbase.client.TestAdmin2 |
| | hadoop.hbase.util.TestHBaseFsck |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint |
| | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA
[jira] [Resolved] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[ https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-21002. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: connector-1.0.0 Release Note: Adds a kafka proxy that appears to hbase as a replication peer. Use to tee table edits to kafka. Has mechanism for dropping/routing updates. See https://github.com/apache/hbase-connectors/tree/master/kafka for documentation. Pushed on hbase-connectors. Was able to stand up the proxy. Looks good. Did not test. Figure we can do follow-on issues. I did a pass on the doc after push to add a bit of formatting and minor edits. Nice one [~MikeWingert] Thanks for the persistence. Let me put a note on dev list... > Create assembly and scripts to start Kafka Proxy > > > Key: HBASE-21002 > URL: https://issues.apache.org/jira/browse/HBASE-21002 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors >Reporter: Mike Wingert >Assignee: Mike Wingert >Priority: Minor > Fix For: connector-1.0.0 > > > Add scripts for running and assembly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.002.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch, > HBASE-21414.master.002.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21416) Intermittent TestRegionInfoDisplay failure due to shift in relTime of RegionState#toDescriptiveString
Ted Yu created HBASE-21416:
--
Summary: Intermittent TestRegionInfoDisplay failure due to shift
in relTime of RegionState#toDescriptiveString
Key: HBASE-21416
URL: https://issues.apache.org/jira/browse/HBASE-21416
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Over
https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2.1/1799/testReport/junit/org.apache.hadoop.hbase.client/TestRegionInfoDisplay/testRegionDetailsForDisplay/
:
{code}
org.junit.ComparisonFailure: expected:<...:30 UTC 2018 (PT0.00[6]S ago),
server=null> but was:<...:30 UTC 2018 (PT0.00[7]S ago), server=null>
at
org.apache.hadoop.hbase.client.TestRegionInfoDisplay.testRegionDetailsForDisplay(TestRegionInfoDisplay.java:78)
{code}
Here is how toDescriptiveString composes relTime:
{code}
long relTime = System.currentTimeMillis() - stamp;
{code}
In the test, state.toDescriptiveString() is called twice for the assertion
where different return values from System.currentTimeMillis() caused the
assertion to fail in the above occasion.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670832#comment-16670832
]
Josh Elser commented on HBASE-21415:
Talked through this with Phil today in Slack.
Best as I can see, we synchronized on the MasterSnapshotManager methods,
through which all snapshot-creation requests flow. My hunch is that this
javadoc is from days-of-old, but didn't dig back into history to confirm.
Added Phil as a contributor and assigned this issue to him to track the
recognition. Thanks again!
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser reassigned HBASE-21415:
--
Assignee: Philippe Laflamme
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Assignee: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670824#comment-16670824
]
Xu Cang commented on HBASE-21415:
-
Nit: besides removing it, maybe also add "the HBase master will simply run them
sequentially " as you mentioned above? Thanks.
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[ https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670819#comment-16670819 ] ASF GitHub Bot commented on HBASE-21002: saintstack commented on issue #3: HBASE-21002 make an assembly for hbase-connectors URL: https://github.com/apache/hbase-connectors/pull/3#issuecomment-434862866 I applied the patch and was able to start up the proxy. I did not test if it worked replicating out to kafka. I figure that can be done in follow-on post-commit. Stuff seems to basically work with basic doc. That is good enough for first commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create assembly and scripts to start Kafka Proxy > > > Key: HBASE-21002 > URL: https://issues.apache.org/jira/browse/HBASE-21002 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors >Reporter: Mike Wingert >Assignee: Mike Wingert >Priority: Minor > > Add scripts for running and assembly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670821#comment-16670821
]
ASF GitHub Bot commented on HBASE-21002:
saintstack closed pull request #3: HBASE-21002 make an assembly for
hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/bin/hbase-connectors b/bin/hbase-connectors
new file mode 100755
index 000..50d1b4f
--- /dev/null
+++ b/bin/hbase-connectors
@@ -0,0 +1,284 @@
+#! /usr/bin/env bash
+#
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+#
+# The hbase command script. Based on the hadoop command script putting
+# in hbase classes, libs and configurations ahead of hadoop's.
+#
+# TODO: Narrow the amount of duplicated code.
+#
+# Environment Variables:
+#
+# JAVA_HOMEThe java implementation to use. Overrides JAVA_HOME.
+# HBASE_CONNECTOR_CLASSPATH_PREFIX Extra Java CLASSPATH entries that should
be
+#prefixed to the system classpath.
+#
+# HBASE_CONNECTOR_HEAPSIZE The maximum amount of heap to use.
+#Default is unset and uses the JVMs default setting
+#(usually 1/4th of the available memory).
+#
+# HBASE_CONNECTOR_LIBRARY_PATH HBase additions to JAVA_LIBRARY_PATH for
adding
+#native libraries.
+#
+# HBASE_CONNECTOR_OPTS Extra Java runtime options.
+#
+# HBASE_CONNECTOR_CONF_DIR Alternate conf dir. Default is
${HBASE_CONNECTOR_HOME}/conf.
+#
+# HBASE_CONNECTOR_ROOT_LOGGER The root appender. Default is INFO,console
+#
+
+
+bin=`dirname "$0"`
+bin=`cd "$bin">/dev/null; pwd`
+
+# This will set HBASE_CONNECTOR_HOME etc.
+. "$bin"/hbase-connectors-config.sh
+
+
+cygwin=false
+case "`uname`" in
+CYGWIN*) cygwin=true;;
+esac
+
+# Detect if we are in hbase sources dir
+in_dev_env=false
+if [ -d "${HBASE_CONNECTOR_HOME}/target" ]; then
+ in_dev_env=true
+fi
+
+# Detect if we are in the omnibus tarball
+in_omnibus_tarball="false"
+if [ -f "${HBASE_CONNECTOR_HOME}/bin/hbase-connectors-daemon.sh" ]; then
+ in_omnibus_tarball="true"
+fi
+
+# if no args specified, show usage
+if [ $# = 0 ]; then
+ echo "Usage: hbase-connectors [] []"
+ echo ""
+ echo "Commands:"
+
+ if [ "${in_omnibus_tarball}" = "true" ]; then
+echo " kafkaproxy Run the HBase Kafka Proxy server"
+echo " kafkaproxytest Run the HBase Kafka Proxy sample kafka listener"
+ fi
+
+ echo " CLASSNAME Run the class named CLASSNAME"
+ exit 1
+fi
+
+# get arguments
+COMMAND=$1
+shift
+
+JAVA=$JAVA_HOME/bin/java
+
+# override default settings for this command, if applicable
+if [ -f "$HBASE_CONNECTOR_HOME/conf/hbase-connector-env-$COMMAND.sh" ]; then
+ . "$HBASE_CONNECTOR_HOME/conf/hbase-connector-env-$COMMAND.sh"
+fi
+
+add_size_suffix() {
+# add an 'm' suffix if the argument is missing one, otherwise use whats
there
+local val="$1"
+local lastchar=${val: -1}
+if [[ "mMgG" == *$lastchar* ]]; then
+echo $val
+else
+echo ${val}m
+fi
+}
+
+
+
+if [[ -n "$HBASE_CONNECTOR_HEAPSIZE" ]]; then
+JAVA_HEAP_MAX="-Xmx$(add_size_suffix $HBASE_CONNECTOR_HEAPSIZE)"
+fi
+
+if [[ -n "$HBASE_CONNECTOR_OFFHEAPSIZE" ]]; then
+JAVA_OFFHEAP_MAX="-XX:MaxDirectMemorySize=$(add_size_suffix
$HBASE_OFFHEAPSIZE)"
+fi
+
+# so that filenames w/ spaces are handled correctly in loops below
+ORIG_IFS=$IFS
+IFS=
+
+# CLASSPATH initially contains $HBASE_CONNECTOR_CONF_DIR
+PASS_CLASSPATH="${HBASE_CONNECTOR_CONF_DIR}"
+
+CLASSPATH=${PASS_CLASSPATH}:$JAVA_HOME/lib/tools.jar
+
+HBASE_IN_PATH=$(which hbase 2>/dev/null)
+
+if [ "$HBASE_IN_PATH" = "" ]; then
+echo "hbase command must be in the path.. aborting"
+exit 1
+fi
+
+# default log directory & file
+if [ "$HBASE_CONNECTOR_LOG_DIR" = "" ]; then
+ HBASE_CONNECTOR_LOG_DIR="$HBASE_CONNECTOR_HOME/logs"
+fi
+if [
[GitHub] saintstack commented on issue #3: HBASE-21002 make an assembly for hbase-connectors
saintstack commented on issue #3: HBASE-21002 make an assembly for hbase-connectors URL: https://github.com/apache/hbase-connectors/pull/3#issuecomment-434862866 I applied the patch and was able to start up the proxy. I did not test if it worked replicating out to kafka. I figure that can be done in follow-on post-commit. Stuff seems to basically work with basic doc. That is good enough for first commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] saintstack closed pull request #3: HBASE-21002 make an assembly for hbase-connectors
saintstack closed pull request #3: HBASE-21002 make an assembly for
hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/bin/hbase-connectors b/bin/hbase-connectors
new file mode 100755
index 000..50d1b4f
--- /dev/null
+++ b/bin/hbase-connectors
@@ -0,0 +1,284 @@
+#! /usr/bin/env bash
+#
+#/**
+# * Licensed to the Apache Software Foundation (ASF) under one
+# * or more contributor license agreements. See the NOTICE file
+# * distributed with this work for additional information
+# * regarding copyright ownership. The ASF licenses this file
+# * to you under the Apache License, Version 2.0 (the
+# * "License"); you may not use this file except in compliance
+# * with the License. You may obtain a copy of the License at
+# *
+# * http://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# */
+#
+# The hbase command script. Based on the hadoop command script putting
+# in hbase classes, libs and configurations ahead of hadoop's.
+#
+# TODO: Narrow the amount of duplicated code.
+#
+# Environment Variables:
+#
+# JAVA_HOMEThe java implementation to use. Overrides JAVA_HOME.
+# HBASE_CONNECTOR_CLASSPATH_PREFIX Extra Java CLASSPATH entries that should
be
+#prefixed to the system classpath.
+#
+# HBASE_CONNECTOR_HEAPSIZE The maximum amount of heap to use.
+#Default is unset and uses the JVMs default setting
+#(usually 1/4th of the available memory).
+#
+# HBASE_CONNECTOR_LIBRARY_PATH HBase additions to JAVA_LIBRARY_PATH for
adding
+#native libraries.
+#
+# HBASE_CONNECTOR_OPTS Extra Java runtime options.
+#
+# HBASE_CONNECTOR_CONF_DIR Alternate conf dir. Default is
${HBASE_CONNECTOR_HOME}/conf.
+#
+# HBASE_CONNECTOR_ROOT_LOGGER The root appender. Default is INFO,console
+#
+
+
+bin=`dirname "$0"`
+bin=`cd "$bin">/dev/null; pwd`
+
+# This will set HBASE_CONNECTOR_HOME etc.
+. "$bin"/hbase-connectors-config.sh
+
+
+cygwin=false
+case "`uname`" in
+CYGWIN*) cygwin=true;;
+esac
+
+# Detect if we are in hbase sources dir
+in_dev_env=false
+if [ -d "${HBASE_CONNECTOR_HOME}/target" ]; then
+ in_dev_env=true
+fi
+
+# Detect if we are in the omnibus tarball
+in_omnibus_tarball="false"
+if [ -f "${HBASE_CONNECTOR_HOME}/bin/hbase-connectors-daemon.sh" ]; then
+ in_omnibus_tarball="true"
+fi
+
+# if no args specified, show usage
+if [ $# = 0 ]; then
+ echo "Usage: hbase-connectors [] []"
+ echo ""
+ echo "Commands:"
+
+ if [ "${in_omnibus_tarball}" = "true" ]; then
+echo " kafkaproxy Run the HBase Kafka Proxy server"
+echo " kafkaproxytest Run the HBase Kafka Proxy sample kafka listener"
+ fi
+
+ echo " CLASSNAME Run the class named CLASSNAME"
+ exit 1
+fi
+
+# get arguments
+COMMAND=$1
+shift
+
+JAVA=$JAVA_HOME/bin/java
+
+# override default settings for this command, if applicable
+if [ -f "$HBASE_CONNECTOR_HOME/conf/hbase-connector-env-$COMMAND.sh" ]; then
+ . "$HBASE_CONNECTOR_HOME/conf/hbase-connector-env-$COMMAND.sh"
+fi
+
+add_size_suffix() {
+# add an 'm' suffix if the argument is missing one, otherwise use whats
there
+local val="$1"
+local lastchar=${val: -1}
+if [[ "mMgG" == *$lastchar* ]]; then
+echo $val
+else
+echo ${val}m
+fi
+}
+
+
+
+if [[ -n "$HBASE_CONNECTOR_HEAPSIZE" ]]; then
+JAVA_HEAP_MAX="-Xmx$(add_size_suffix $HBASE_CONNECTOR_HEAPSIZE)"
+fi
+
+if [[ -n "$HBASE_CONNECTOR_OFFHEAPSIZE" ]]; then
+JAVA_OFFHEAP_MAX="-XX:MaxDirectMemorySize=$(add_size_suffix
$HBASE_OFFHEAPSIZE)"
+fi
+
+# so that filenames w/ spaces are handled correctly in loops below
+ORIG_IFS=$IFS
+IFS=
+
+# CLASSPATH initially contains $HBASE_CONNECTOR_CONF_DIR
+PASS_CLASSPATH="${HBASE_CONNECTOR_CONF_DIR}"
+
+CLASSPATH=${PASS_CLASSPATH}:$JAVA_HOME/lib/tools.jar
+
+HBASE_IN_PATH=$(which hbase 2>/dev/null)
+
+if [ "$HBASE_IN_PATH" = "" ]; then
+echo "hbase command must be in the path.. aborting"
+exit 1
+fi
+
+# default log directory & file
+if [ "$HBASE_CONNECTOR_LOG_DIR" = "" ]; then
+ HBASE_CONNECTOR_LOG_DIR="$HBASE_CONNECTOR_HOME/logs"
+fi
+if [ "$HBASE_CONNECTOR_LOGFILE" = "" ]; then
+ HBASE_CONNECTOR_LOGFILE='hbase-connector.log'
+fi
+
+function append_path() {
+ if [ -z "$1" ]; then
+echo "$2"
+ else
+echo "$1:$2"
+ fi
+}
+
+JAVA_PLATFORM=""
+
+# if HBASE_CONNECTOR_LIBRARY_PATH is
[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670817#comment-16670817
]
ASF GitHub Bot commented on HBASE-21002:
saintstack commented on a change in pull request #3: HBASE-21002 make an
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229882057
##
File path: kafka/hbase-kafka-proxy/pom.xml
##
@@ -76,43 +77,40 @@
org.apache.avro
avro
-
+
org.apache.hbase.connectors
hbase-kafka-model
-
- org.apache.hbase
- hbase-common
-
+
org.apache.hbase
hbase-common
test-jar
test
-
- org.apache.hbase
- hbase-client
-
-
- org.apache.hbase
- hbase-zookeeper
-
+
org.apache.hbase
hbase-server
+ provided
org.apache.hbase
hbase-annotations
+ compile
org.apache.hbase
hbase-annotations
test-jar
test
+
+ org.apache.yetus
+ audience-annotations
+ ${audience-annotations.version}
+
org.apache.kafka
kafka-clients
Review comment:
Ok. Makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create assembly and scripts to start Kafka Proxy
>
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] saintstack commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors
saintstack commented on a change in pull request #3: HBASE-21002 make an
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229882057
##
File path: kafka/hbase-kafka-proxy/pom.xml
##
@@ -76,43 +77,40 @@
org.apache.avro
avro
-
+
org.apache.hbase.connectors
hbase-kafka-model
-
- org.apache.hbase
- hbase-common
-
+
org.apache.hbase
hbase-common
test-jar
test
-
- org.apache.hbase
- hbase-client
-
-
- org.apache.hbase
- hbase-zookeeper
-
+
org.apache.hbase
hbase-server
+ provided
org.apache.hbase
hbase-annotations
+ compile
org.apache.hbase
hbase-annotations
test-jar
test
+
+ org.apache.yetus
+ audience-annotations
+ ${audience-annotations.version}
+
org.apache.kafka
kafka-clients
Review comment:
Ok. Makes sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Assigned] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-21381:
--
Assignee: (was: Ted Yu)
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670811#comment-16670811
]
Ted Yu commented on HBASE-21381:
Putting the supported version under
http://hbase.apache.org/book.html#backuprestore is fine.
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21381) Document the hadoop versions using which backup and restore feature works
[
https://issues.apache.org/jira/browse/HBASE-21381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-21381:
--
Assignee: Ted Yu
> Document the hadoop versions using which backup and restore feature works
> -
>
> Key: HBASE-21381
> URL: https://issues.apache.org/jira/browse/HBASE-21381
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
>
> HADOOP-15850 fixes a bug where CopyCommitter#concatFileChunks unconditionally
> tried to concatenate the files being DistCp'ed to target cluster (though the
> files are independent).
> Following is the log snippet of the failed concatenation attempt:
> {code}
> 2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
> job_local1795473782_0004
> java.io.IOException: Inconsistent sequence file: current chunk file
> org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
>
> 160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
> length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
> org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
>
> 2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
> length = 5142 aclEntries = null, xAttrs = null}
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
> at
> org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
> {code}
> Backup and Restore uses DistCp to transfer files between clusters.
> Without the fix from HADOOP-15850, the transfer would fail.
> This issue is to document the hadoop versions which contain HADOOP-15850 so
> that user of Backup and Restore feature knows which hadoop versions they can
> use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21415) Admin#snapshot method documentation is misleading.
[
https://issues.apache.org/jira/browse/HBASE-21415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Laflamme updated HBASE-21415:
--
Attachment: HBASE-21415.patch
Status: Patch Available (was: Open)
> Admin#snapshot method documentation is misleading.
> --
>
> Key: HBASE-21415
> URL: https://issues.apache.org/jira/browse/HBASE-21415
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Reporter: Philippe Laflamme
>Priority: Trivial
> Attachments: HBASE-21415.patch
>
>
> The documentation for the {{Admin#snapshot}} function is misleading in
> regards to snapshot concurrency.
> It currently states the following:
> {quote}Only a single snapshot should be taken at a time for an instance of
> HBase, or results may be undefined (you can tell multiple HBase clusters to
> snapshot at the same time, but only one at a time for a single cluster).
> {quote}
> This seems to indicate that it's dangerous to run snapshots concurrently when
> in fact the HBase master will simply run them sequentially (as per this Slack
> thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
> The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21415) Admin#snapshot method documentation is misleading.
Philippe Laflamme created HBASE-21415:
-
Summary: Admin#snapshot method documentation is misleading.
Key: HBASE-21415
URL: https://issues.apache.org/jira/browse/HBASE-21415
Project: HBase
Issue Type: Improvement
Components: documentation
Reporter: Philippe Laflamme
The documentation for the {{Admin#snapshot}} function is misleading in regards
to snapshot concurrency.
It currently states the following:
{quote}Only a single snapshot should be taken at a time for an instance of
HBase, or results may be undefined (you can tell multiple HBase clusters to
snapshot at the same time, but only one at a time for a single cluster).
{quote}
This seems to indicate that it's dangerous to run snapshots concurrently when
in fact the HBase master will simply run them sequentially (as per this Slack
thread [https://apache-hbase.slack.com/archives/C13K8NVAM/p1540994948005900)]
The suggested fix is to simply remove this entire sentence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21414) StoreFileSize growth rate metric
[
https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670779#comment-16670779
]
Xu Cang commented on HBASE-21414:
-
Nit:
{code:java}
storeFileSizeGrowthRate = (double)intervalStoreFileSize * 1000.0 /
(double)period;
{code}
Redundant (double) here. only need one of them.
> StoreFileSize growth rate metric
>
>
> Key: HBASE-21414
> URL: https://issues.apache.org/jira/browse/HBASE-21414
> Project: HBase
> Issue Type: Improvement
> Components: metrics, monitoring
>Reporter: Tommy Li
>Priority: Minor
> Attachments: HBASE-21414.master.001.patch
>
>
> A metric on the growth rate of storefile sizes would be nice to have as a way
> of monitoring traffic patterns. I know you can get the same insight from
> graphing the delta on the storeFileSize metric, but not all metrics
> visualization tools support that
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670725#comment-16670725
]
ASF GitHub Bot commented on HBASE-21002:
hbasejanitor commented on a change in pull request #3: HBASE-21002 make an
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229867217
##
File path: kafka/hbase-kafka-proxy/pom.xml
##
@@ -76,43 +77,40 @@
org.apache.avro
avro
-
+
org.apache.hbase.connectors
hbase-kafka-model
-
- org.apache.hbase
- hbase-common
-
+
org.apache.hbase
hbase-common
test-jar
test
-
- org.apache.hbase
- hbase-client
-
-
- org.apache.hbase
- hbase-zookeeper
-
+
org.apache.hbase
hbase-server
+ provided
org.apache.hbase
hbase-annotations
+ compile
org.apache.hbase
hbase-annotations
test-jar
test
+
+ org.apache.yetus
+ audience-annotations
+ ${audience-annotations.version}
+
org.apache.kafka
kafka-clients
Review comment:
Sorry, I've been away from this patch for a while.
hbase-client isn't need at all, but hbase-server is. Without it, I don't
have access to the classes to start a region server.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create assembly and scripts to start Kafka Proxy
>
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hbasejanitor commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors
hbasejanitor commented on a change in pull request #3: HBASE-21002 make an
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229867217
##
File path: kafka/hbase-kafka-proxy/pom.xml
##
@@ -76,43 +77,40 @@
org.apache.avro
avro
-
+
org.apache.hbase.connectors
hbase-kafka-model
-
- org.apache.hbase
- hbase-common
-
+
org.apache.hbase
hbase-common
test-jar
test
-
- org.apache.hbase
- hbase-client
-
-
- org.apache.hbase
- hbase-zookeeper
-
+
org.apache.hbase
hbase-server
+ provided
org.apache.hbase
hbase-annotations
+ compile
org.apache.hbase
hbase-annotations
test-jar
test
+
+ org.apache.yetus
+ audience-annotations
+ ${audience-annotations.version}
+
org.apache.kafka
kafka-clients
Review comment:
Sorry, I've been away from this patch for a while.
hbase-client isn't need at all, but hbase-server is. Without it, I don't
have access to the classes to start a region server.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[
https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670704#comment-16670704
]
ASF GitHub Bot commented on HBASE-21002:
saintstack commented on a change in pull request #3: HBASE-21002 make an
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229864093
##
File path: kafka/hbase-kafka-proxy/pom.xml
##
@@ -76,43 +77,40 @@
org.apache.avro
avro
-
+
org.apache.hbase.connectors
hbase-kafka-model
-
- org.apache.hbase
- hbase-common
-
+
org.apache.hbase
hbase-common
test-jar
test
-
- org.apache.hbase
- hbase-client
-
-
- org.apache.hbase
- hbase-zookeeper
-
+
org.apache.hbase
hbase-server
+ provided
org.apache.hbase
hbase-annotations
+ compile
org.apache.hbase
hbase-annotations
test-jar
test
+
+ org.apache.yetus
+ audience-annotations
+ ${audience-annotations.version}
+
org.apache.kafka
kafka-clients
Review comment:
This going to be done in a follow-up?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Create assembly and scripts to start Kafka Proxy
>
>
> Key: HBASE-21002
> URL: https://issues.apache.org/jira/browse/HBASE-21002
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
>Reporter: Mike Wingert
>Assignee: Mike Wingert
>Priority: Minor
>
> Add scripts for running and assembly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21002) Create assembly and scripts to start Kafka Proxy
[ https://issues.apache.org/jira/browse/HBASE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670706#comment-16670706 ] ASF GitHub Bot commented on HBASE-21002: saintstack commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229864154 ## File path: kafka/README ## @@ -0,0 +1,126 @@ +Hbase Kafka Proxy Review comment: Duh This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Create assembly and scripts to start Kafka Proxy > > > Key: HBASE-21002 > URL: https://issues.apache.org/jira/browse/HBASE-21002 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors >Reporter: Mike Wingert >Assignee: Mike Wingert >Priority: Minor > > Add scripts for running and assembly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] saintstack commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors
saintstack commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229864154 ## File path: kafka/README ## @@ -0,0 +1,126 @@ +Hbase Kafka Proxy Review comment: Duh This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] saintstack commented on a change in pull request #3: HBASE-21002 make an assembly for hbase-connectors
saintstack commented on a change in pull request #3: HBASE-21002 make an
assembly for hbase-connectors
URL: https://github.com/apache/hbase-connectors/pull/3#discussion_r229864093
##
File path: kafka/hbase-kafka-proxy/pom.xml
##
@@ -76,43 +77,40 @@
org.apache.avro
avro
-
+
org.apache.hbase.connectors
hbase-kafka-model
-
- org.apache.hbase
- hbase-common
-
+
org.apache.hbase
hbase-common
test-jar
test
-
- org.apache.hbase
- hbase-client
-
-
- org.apache.hbase
- hbase-zookeeper
-
+
org.apache.hbase
hbase-server
+ provided
org.apache.hbase
hbase-annotations
+ compile
org.apache.hbase
hbase-annotations
test-jar
test
+
+ org.apache.yetus
+ audience-annotations
+ ${audience-annotations.version}
+
org.apache.kafka
kafka-clients
Review comment:
This going to be done in a follow-up?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: (was: HBASE-21414.master.002.patch) > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.002.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch, > HBASE-21414.master.002.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21414) StoreFileSize growth rate metric
[ https://issues.apache.org/jira/browse/HBASE-21414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21414: - Attachment: HBASE-21414.master.001.patch > StoreFileSize growth rate metric > > > Key: HBASE-21414 > URL: https://issues.apache.org/jira/browse/HBASE-21414 > Project: HBase > Issue Type: Improvement > Components: metrics, monitoring >Reporter: Tommy Li >Priority: Minor > Attachments: HBASE-21414.master.001.patch > > > A metric on the growth rate of storefile sizes would be nice to have as a way > of monitoring traffic patterns. I know you can get the same insight from > graphing the delta on the storeFileSize metric, but not all metrics > visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21414) StoreFileSize growth rate metric
Tommy Li created HBASE-21414: Summary: StoreFileSize growth rate metric Key: HBASE-21414 URL: https://issues.apache.org/jira/browse/HBASE-21414 Project: HBase Issue Type: Improvement Components: metrics, monitoring Reporter: Tommy Li A metric on the growth rate of storefile sizes would be nice to have as a way of monitoring traffic patterns. I know you can get the same insight from graphing the delta on the storeFileSize metric, but not all metrics visualization tools support that -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21374) Backport HBASE-21342 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21374: -- Attachment: HBASE-21374.branch-1.003.patch > Backport HBASE-21342 to branch-1 > > > Key: HBASE-21374 > URL: https://issues.apache.org/jira/browse/HBASE-21374 > Project: HBase > Issue Type: Task >Reporter: Mike Drob >Assignee: mazhenlin >Priority: Major > Attachments: HBASE-21374.branch-1.001.patch, > HBASE-21374.branch-1.002.patch, HBASE-21374.branch-1.003.patch, > HBASE-21374.branch-1.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21374) Backport HBASE-21342 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670629#comment-16670629 ] stack commented on HBASE-21374: --- Retry > Backport HBASE-21342 to branch-1 > > > Key: HBASE-21374 > URL: https://issues.apache.org/jira/browse/HBASE-21374 > Project: HBase > Issue Type: Task >Reporter: Mike Drob >Assignee: mazhenlin >Priority: Major > Attachments: HBASE-21374.branch-1.001.patch, > HBASE-21374.branch-1.002.patch, HBASE-21374.branch-1.003.patch, > HBASE-21374.branch-1.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21191) Add a holding-pattern if no assign for meta or namespace (Can happen if masterprocwals have been cleared).
[ https://issues.apache.org/jira/browse/HBASE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21191: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed https://issues.apache.org/jira/secure/attachment/12945831/HBASE-21191.branch-2.0.001.patch to branch-2.0. Closing now this is in 2.1.1 and 2.0.3. > Add a holding-pattern if no assign for meta or namespace (Can happen if > masterprocwals have been cleared). > -- > > Key: HBASE-21191 > URL: https://issues.apache.org/jira/browse/HBASE-21191 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21191.branch-2.0.001.patch, > HBASE-21191.branch-2.1.001.patch, HBASE-21191.branch-2.1.002.patch, > HBASE-21191.branch-2.1.003.patch, HBASE-21191.branch-2.1.004.patch, > HBASE-21191.branch-2.1.005.patch, HBASE-21191.branch-2.1.006.patch, > HBASE-21191.branch-2.1.007.patch > > > If the masterprocwals have been removed -- operator error, hdfs dataloss, or > because we have gotten ourselves into a pathological state where we have > hundreds of masterprocwals too process and it is taking too long so we just > want to startover -- then master startup will have a dilemma. Master startup > needs hbase:meta to be online. If the masterprocwals have been removed, there > may be no outstanding assign or a servercrashprocedure with coverage for > hbase:meta (I ran into this issue repeatedly in internal testing purging > masterprocwals on a large test cluster). Worse, when master startup cannot > find an online hbase:meta, it exits after exhausting the RPC retries. > So, we need a holding-pattern for master startup if hbase:meta is not online > if only so an operator can schedule an assign for meta or so they can assign > fixup procedures (HBASE-21035 has discussion on why we cannot just > auto-schedule an assign of meta). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21408) Add hbase.wal.dir clean operation also to hbase cleanup script.
[ https://issues.apache.org/jira/browse/HBASE-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21408: -- Fix Version/s: (was: 2.1.1) 2.1.2 > Add hbase.wal.dir clean operation also to hbase cleanup script. > --- > > Key: HBASE-21408 > URL: https://issues.apache.org/jira/browse/HBASE-21408 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.1.0 >Reporter: Y. SREENIVASULU REDDY >Priority: Major > Fix For: 2.1.2 > > Attachments: HBASE-21408.001.patch > > > If user configured hbase.wal.dir explicitly. > cleaning scripts should handle this too. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670566#comment-16670566 ] Mike Wingert commented on HBASE-15320: -- Hi Stack, the proxy moved to the hbase-connectors project. Can you look at the pull request at https://issues.apache.org/jira/browse/HBASE-21002? Once that's in, the proxy in the hbase-connectors project will be working. > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert >Priority: Major > Labels: beginner > Fix For: 3.0.0 > > Attachments: 15320.master.16.patch, 15320.master.16.patch, > HBASE-15320.master.1.patch, HBASE-15320.master.10.patch, > HBASE-15320.master.11.patch, HBASE-15320.master.12.patch, > HBASE-15320.master.14.patch, HBASE-15320.master.15.patch, > HBASE-15320.master.2.patch, HBASE-15320.master.3.patch, > HBASE-15320.master.4.patch, HBASE-15320.master.5.patch, > HBASE-15320.master.6.patch, HBASE-15320.master.7.patch, > HBASE-15320.master.8.patch, HBASE-15320.master.8.patch, > HBASE-15320.master.9.patch, HBASE-15320.pdf, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[
https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670551#comment-16670551
]
Hadoop QA commented on HBASE-15320:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-15320 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-15320 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932784/15320.master.16.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14911/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> HBase connector for Kafka Connect
> -
>
> Key: HBASE-15320
> URL: https://issues.apache.org/jira/browse/HBASE-15320
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Andrew Purtell
>Assignee: Mike Wingert
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0
>
> Attachments: 15320.master.16.patch, 15320.master.16.patch,
> HBASE-15320.master.1.patch, HBASE-15320.master.10.patch,
> HBASE-15320.master.11.patch, HBASE-15320.master.12.patch,
> HBASE-15320.master.14.patch, HBASE-15320.master.15.patch,
> HBASE-15320.master.2.patch, HBASE-15320.master.3.patch,
> HBASE-15320.master.4.patch, HBASE-15320.master.5.patch,
> HBASE-15320.master.6.patch, HBASE-15320.master.7.patch,
> HBASE-15320.master.8.patch, HBASE-15320.master.8.patch,
> HBASE-15320.master.9.patch, HBASE-15320.pdf, HBASE-15320.pdf
>
>
> Implement an HBase connector with source and sink tasks for the Connect
> framework (http://docs.confluent.io/2.0.0/connect/index.html) available in
> Kafka 0.9 and later.
> See also:
> http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines
> An HBase source
> (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task)
> could be implemented as a replication endpoint or WALObserver, publishing
> cluster wide change streams from the WAL to one or more topics, with
> configurable mapping and partitioning of table changes to topics.
> An HBase sink task
> (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would
> persist, with optional transformation (JSON? Avro?, map fields to native
> schema?), Kafka SinkRecords into HBase tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670538#comment-16670538 ] stack commented on HBASE-15320: --- [~MikeWingert] doing > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert >Priority: Major > Labels: beginner > Fix For: 3.0.0 > > Attachments: 15320.master.16.patch, 15320.master.16.patch, > HBASE-15320.master.1.patch, HBASE-15320.master.10.patch, > HBASE-15320.master.11.patch, HBASE-15320.master.12.patch, > HBASE-15320.master.14.patch, HBASE-15320.master.15.patch, > HBASE-15320.master.2.patch, HBASE-15320.master.3.patch, > HBASE-15320.master.4.patch, HBASE-15320.master.5.patch, > HBASE-15320.master.6.patch, HBASE-15320.master.7.patch, > HBASE-15320.master.8.patch, HBASE-15320.master.8.patch, > HBASE-15320.master.9.patch, HBASE-15320.pdf, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670530#comment-16670530 ] Mike Wingert commented on HBASE-15320: -- Hi Stack, not a problem. Can you try my patch at https://issues.apache.org/jira/browse/HBASE-21002 and see if you can build & run the proxy? If that works for you, I think the initial pass of this Jira is done. > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert >Priority: Major > Labels: beginner > Fix For: 3.0.0 > > Attachments: 15320.master.16.patch, 15320.master.16.patch, > HBASE-15320.master.1.patch, HBASE-15320.master.10.patch, > HBASE-15320.master.11.patch, HBASE-15320.master.12.patch, > HBASE-15320.master.14.patch, HBASE-15320.master.15.patch, > HBASE-15320.master.2.patch, HBASE-15320.master.3.patch, > HBASE-15320.master.4.patch, HBASE-15320.master.5.patch, > HBASE-15320.master.6.patch, HBASE-15320.master.7.patch, > HBASE-15320.master.8.patch, HBASE-15320.master.8.patch, > HBASE-15320.master.9.patch, HBASE-15320.pdf, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15320) HBase connector for Kafka Connect
[ https://issues.apache.org/jira/browse/HBASE-15320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670520#comment-16670520 ] stack commented on HBASE-15320: --- What do I need to do here [~MikeWingert] ? I dropped the ball sir. > HBase connector for Kafka Connect > - > > Key: HBASE-15320 > URL: https://issues.apache.org/jira/browse/HBASE-15320 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: Andrew Purtell >Assignee: Mike Wingert >Priority: Major > Labels: beginner > Fix For: 3.0.0 > > Attachments: 15320.master.16.patch, 15320.master.16.patch, > HBASE-15320.master.1.patch, HBASE-15320.master.10.patch, > HBASE-15320.master.11.patch, HBASE-15320.master.12.patch, > HBASE-15320.master.14.patch, HBASE-15320.master.15.patch, > HBASE-15320.master.2.patch, HBASE-15320.master.3.patch, > HBASE-15320.master.4.patch, HBASE-15320.master.5.patch, > HBASE-15320.master.6.patch, HBASE-15320.master.7.patch, > HBASE-15320.master.8.patch, HBASE-15320.master.8.patch, > HBASE-15320.master.9.patch, HBASE-15320.pdf, HBASE-15320.pdf > > > Implement an HBase connector with source and sink tasks for the Connect > framework (http://docs.confluent.io/2.0.0/connect/index.html) available in > Kafka 0.9 and later. > See also: > http://www.confluent.io/blog/announcing-kafka-connect-building-large-scale-low-latency-data-pipelines > An HBase source > (http://docs.confluent.io/2.0.0/connect/devguide.html#task-example-source-task) > could be implemented as a replication endpoint or WALObserver, publishing > cluster wide change streams from the WAL to one or more topics, with > configurable mapping and partitioning of table changes to topics. > An HBase sink task > (http://docs.confluent.io/2.0.0/connect/devguide.html#sink-tasks) would > persist, with optional transformation (JSON? Avro?, map fields to native > schema?), Kafka SinkRecords into HBase tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21406) "status 'replication'" should not show SINK if the cluster does not act as sink
[
https://issues.apache.org/jira/browse/HBASE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670490#comment-16670490
]
Wellington Chevreuil commented on HBASE-21406:
--
Issue here is that ReplicationSink is always initialised as part of the
replication services:
!Screen Shot 2018-10-31 at 18.12.54.png!
It's then kept by HRegionServer instance and only invoked once some RPC to
replicateWalEntry method is invoked. It has the metrics, but it's not supposed
to know if something should be replicating to it or not. One alternative here
would be to change *MetricsReplicationSinkSource,* to track/expose specific
timestamp attribute for the sink initialisation. That way, clients/UIs could
use it to compare with timestampOfLastApplied and define if there's ever any OP
shipped. Another interesting info to expose on status command output is the
total amount of applied OPs (which is already tracked and showed via JMX).
I will be proposing an initial patch with this approach later.
> "status 'replication'" should not show SINK if the cluster does not act as
> sink
> ---
>
> Key: HBASE-21406
> URL: https://issues.apache.org/jira/browse/HBASE-21406
> Project: HBase
> Issue Type: Improvement
>Reporter: Daisuke Kobayashi
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: Screen Shot 2018-10-31 at 18.12.54.png
>
>
> When replicating in 1 way, from source to target, {{status 'replication'}} on
> source always dumps SINK with meaningless metrics. It only makes sense when
> running the command on target cluster.
> {{status 'replication'}} on source, for example. {{AgeOfLastAppliedOp}} is
> always zero and {{TimeStampsOfLastAppliedOp}} does not get updated from the
> time the RS started since it's not acting as sink.
> {noformat}
> source-1.com
>SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=0,
> TimeStampsOfLastShippedOp=Mon Oct 29 23:44:14 PDT 2018, Replication Lag=0
>SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Oct 25
> 23:56:53 PDT 2018
> {noformat}
> {{status 'replication'}} on target works as expected. SOURCE is empty as it's
> not acting as source:
> {noformat}
> target-1.com
>SOURCE:
>SINK : AgeOfLastAppliedOp=70, TimeStampsOfLastAppliedOp=Mon Oct 29
> 23:44:08 PDT 2018
> {noformat}
> This is because {{getReplicationLoadSink}}, called in {{admin.rb}}, always
> returns a value (not null).
> 1.X
> https://github.com/apache/hbase/blob/rel/1.4.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L194-L204
> 2.X
> https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L392-L399
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21406) "status 'replication'" should not show SINK if the cluster does not act as sink
[
https://issues.apache.org/jira/browse/HBASE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-21406:
-
Attachment: Screen Shot 2018-10-31 at 18.12.54.png
> "status 'replication'" should not show SINK if the cluster does not act as
> sink
> ---
>
> Key: HBASE-21406
> URL: https://issues.apache.org/jira/browse/HBASE-21406
> Project: HBase
> Issue Type: Improvement
>Reporter: Daisuke Kobayashi
>Assignee: Wellington Chevreuil
>Priority: Minor
> Attachments: Screen Shot 2018-10-31 at 18.12.54.png
>
>
> When replicating in 1 way, from source to target, {{status 'replication'}} on
> source always dumps SINK with meaningless metrics. It only makes sense when
> running the command on target cluster.
> {{status 'replication'}} on source, for example. {{AgeOfLastAppliedOp}} is
> always zero and {{TimeStampsOfLastAppliedOp}} does not get updated from the
> time the RS started since it's not acting as sink.
> {noformat}
> source-1.com
>SOURCE: PeerID=1, AgeOfLastShippedOp=0, SizeOfLogQueue=0,
> TimeStampsOfLastShippedOp=Mon Oct 29 23:44:14 PDT 2018, Replication Lag=0
>SINK : AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Thu Oct 25
> 23:56:53 PDT 2018
> {noformat}
> {{status 'replication'}} on target works as expected. SOURCE is empty as it's
> not acting as source:
> {noformat}
> target-1.com
>SOURCE:
>SINK : AgeOfLastAppliedOp=70, TimeStampsOfLastAppliedOp=Mon Oct 29
> 23:44:08 PDT 2018
> {noformat}
> This is because {{getReplicationLoadSink}}, called in {{admin.rb}}, always
> returns a value (not null).
> 1.X
> https://github.com/apache/hbase/blob/rel/1.4.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L194-L204
> 2.X
> https://github.com/apache/hbase/blob/rel/2.0.0/hbase-client/src/main/java/org/apache/hadoop/hbase/ServerLoad.java#L392-L399
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670472#comment-16670472 ] Ted Yu commented on HBASE-21246: wal-providers.png is diagram for WALProvider related classes. wal-factory-providers.png is diagram involving WALProvider related classes and WALFactory class. > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21246: --- Attachment: wal-providers.png wal-factory-providers.png > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch, > wal-factory-providers.png, wal-providers.png > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21301) Heatmap for key access patterns
[ https://issues.apache.org/jira/browse/HBASE-21301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670427#comment-16670427 ] stack commented on HBASE-21301: --- Try removing all old data... If in standalone mode, see what the path to your temporary directory is... its something like: Server environment:java.io.tmpdir=/var/folders/d8/8lyxycpd129d4fj7lb684dwhgp/T/ Remove the hbase dir under there and the zookeeper one for good measure after killing all daemons. If a recent master, you could also inject an assign of the namespace region but above is probably easier on you. > Heatmap for key access patterns > --- > > Key: HBASE-21301 > URL: https://issues.apache.org/jira/browse/HBASE-21301 > Project: HBase > Issue Type: Improvement >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21301.v1.patch > > > Google recently released a beta feature for Cloud Bigtable which presents a > heat map of the keyspace. *Given how hotspotting comes up now and again here, > this is a good idea for giving HBase ops a tool to be proactive about it.* > >>> > Additionally, we are announcing the beta version of Key Visualizer, a > visualization tool for Cloud Bigtable key access patterns. Key Visualizer > helps debug performance issues due to unbalanced access patterns across the > key space, or single rows that are too large or receiving too much read or > write activity. With Key Visualizer, you get a heat map visualization of > access patterns over time, along with the ability to zoom into specific key > or time ranges, or select a specific row to find the full row key ID that's > responsible for a hotspot. Key Visualizer is automatically enabled for Cloud > Bigtable clusters with sufficient data or activity, and does not affect Cloud > Bigtable cluster performance. > <<< > From > [https://cloudplatform.googleblog.com/2018/07/on-gcp-your-database-your-way.html] > (Copied this description from the write-up by [~apurtell], thanks Andrew.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21301) Heatmap for key access patterns
[
https://issues.apache.org/jira/browse/HBASE-21301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670356#comment-16670356
]
Archana Katiyar commented on HBASE-21301:
-
Need help. When I am testing changes for this Jira using 'hbase master start',
I am seeing following error -
_2018-10-31 21:58:46,444 WARN [master/10.0.0.8:16000:becomeActiveMaster]
master.HMaster:
hbase:namespace,,1540714799202.4877e41f08ffe48fa36526c157d11c40. is NOT online;
state=\{4877e41f08ffe48fa36526c157d11c40 state=OPEN, ts=1541003319363,
server=10.0.0.8,16020,1540714789505}; ServerCrashProcedures=true. Master
startup cannot progress, in holding-pattern until region onlined._
Any idea what is missing here? I am working on master branch.
> Heatmap for key access patterns
> ---
>
> Key: HBASE-21301
> URL: https://issues.apache.org/jira/browse/HBASE-21301
> Project: HBase
> Issue Type: Improvement
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21301.v1.patch
>
>
> Google recently released a beta feature for Cloud Bigtable which presents a
> heat map of the keyspace. *Given how hotspotting comes up now and again here,
> this is a good idea for giving HBase ops a tool to be proactive about it.*
> >>>
> Additionally, we are announcing the beta version of Key Visualizer, a
> visualization tool for Cloud Bigtable key access patterns. Key Visualizer
> helps debug performance issues due to unbalanced access patterns across the
> key space, or single rows that are too large or receiving too much read or
> write activity. With Key Visualizer, you get a heat map visualization of
> access patterns over time, along with the ability to zoom into specific key
> or time ranges, or select a specific row to find the full row key ID that's
> responsible for a hotspot. Key Visualizer is automatically enabled for Cloud
> Bigtable clusters with sufficient data or activity, and does not affect Cloud
> Bigtable cluster performance.
> <<<
> From
> [https://cloudplatform.googleblog.com/2018/07/on-gcp-your-database-your-way.html]
> (Copied this description from the write-up by [~apurtell], thanks Andrew.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21371) Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license error
[
https://issues.apache.org/jira/browse/HBASE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670345#comment-16670345
]
Hudson commented on HBASE-21371:
Results for branch master
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/576/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Hbase unable to compile against Hadoop trunk (3.3.0-SNAPSHOT) due to license
> error
> --
>
> Key: HBASE-21371
> URL: https://issues.apache.org/jira/browse/HBASE-21371
> Project: HBase
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2
>
> Attachments: HBASE-21371.master.001.patch
>
>
> Hadoop 3.3.0 (trunk) updated various dependencies, which adds additional
> licenses that break HBase's license check plugin.
> CDDL/GPLv2+CE license
> {quote}This product includes JavaBeans Activation Framework API jar licensed
> under the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
> ERROR: Please check this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
> javax.activation-api
> 1.2.0
> maven central search
> g:javax.activation AND a:javax.activation-api AND v:1.2.0
> project website
> [http://java.net/all/javax.activation-api/]
> project source
> [https://github.com/javaee/activation/javax.activation-api]
> {quote}
> Bouncy Castle License
> {quote}–
> This product includes Bouncy Castle PKIX, CMS, EAC, TSP, PKCS, OCSP, CMP,
> and CRMF APIs licensed under the Bouncy Castle Licence.
> ERROR: Please check this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.bouncycastle
> bcpkix-jdk15on
> 1.60
> maven central search
> g:org.bouncycastle AND a:bcpkix-jdk15on AND v:1.60
> project website
> [http://www.bouncycastle.org/java.html]
> project source
> [https://github.com/bcgit/bc-java]
> –
> {quote}
>
> And a long list of "Apache Software License - Version 2.0" licensed Jetty
> dependencies like this:
> {quote}
> This product includes Jetty :: Servlet Annotations licensed under the Apache
> Software License - Version 2.0.
> ERROR: Please check this License for acceptability here:
> [https://www.apache.org/legal/resolved]
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> org.eclipse.jetty
> jetty-annotations
> 9.3.19.v20170502
> maven central search
> g:org.eclipse.jetty AND a:jetty-annotations AND v:9.3.19.v20170502
> project website
> [http://www.eclipse.org/jetty]
> project source
> [https://github.com/eclipse/jetty.project/jetty-annotations]
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21322) Add a scheduleServerCrashProcedure() API to HbckService
[
https://issues.apache.org/jira/browse/HBASE-21322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670346#comment-16670346
]
Hudson commented on HBASE-21322:
Results for branch master
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/576/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add a scheduleServerCrashProcedure() API to HbckService
> ---
>
> Key: HBASE-21322
> URL: https://issues.apache.org/jira/browse/HBASE-21322
> Project: HBase
> Issue Type: Sub-task
> Components: hbck2
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2
>
> Attachments: HBASE-21322.branch-2.1.001.patch,
> HBASE-21322.master.001.patch, HBASE-21322.master.002.patch,
> HBASE-21322.master.003.patch, HBASE-21322.master.004.patch,
> HBASE-21322.master.005.patch, HBASE-21322.master.006.patch, Screenshot from
> 2018-10-17 13-35-58.png, Screenshot from 2018-10-17 13-38-41.png, Screenshot
> from 2018-10-17 13-47-06.png
>
>
> According to my test, if one RS is down, then all procedure logs are deleted,
> it will lead to that no ServerCrashProcedure is scheduled. And restarting
> master cannot help. Thus we need to schedule a ServerCrashProcedure manually
> to solve the problem. I plan to add a scheduleServerCrashProcedure() API to
> HbckService, then add this API to HBCK2.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21375) Revisit the lock and queue implementation in MasterProcedureScheduler
[
https://issues.apache.org/jira/browse/HBASE-21375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670344#comment-16670344
]
Hudson commented on HBASE-21375:
Results for branch master
[build #576 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/576/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/576//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Revisit the lock and queue implementation in MasterProcedureScheduler
> -
>
> Key: HBASE-21375
> URL: https://issues.apache.org/jira/browse/HBASE-21375
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.0.3, 2.1.2
>
> Attachments: HBASE-21375-UT.patch, HBASE-21375-UT2.patch,
> HBASE-21375-v1.patch, HBASE-21375-v2.patch, HBASE-21375.patch
>
>
> The problem for the old implementation is that we will only check the first
> procedure in a queue to see if it could run, if it can not, we will remove
> the queue from run queue. So when adding procedure to the scheduler, we have
> to try hard to put the procedure which can be executed in front of the queue,
> if there are corner cases where we fail to do so, it will likely lead to a
> dead lock, that's why we have the tricky code when loading procedures and try
> to add them into the scheduler, and also lots of 'if' in the doAdd method of
> MasterProcedureScheduler. But this is still not enough to make things right,
> so finally [~allan163] and I decided to change the logic in doPoll method,
> where we use a loop to find whether there is a procedure can be executed, not
> only the first one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21237) Use CompatRemoteProcedureResolver to dispatch open/close region requests to RS
[
https://issues.apache.org/jira/browse/HBASE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670322#comment-16670322
]
Hudson commented on HBASE-21237:
Results for branch branch-2.1
[build #558 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/558/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/558//console].
> Use CompatRemoteProcedureResolver to dispatch open/close region requests to RS
> --
>
> Key: HBASE-21237
> URL: https://issues.apache.org/jira/browse/HBASE-21237
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0, 2.0.2
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Blocker
> Fix For: 2.0.3, 2.1.2
>
> Attachments: HBASE-21237-branch-2.1-v1.patch,
> HBASE-21237-branch-2.1.patch, HBASE-21237.branch-2.0.001.patch
>
>
> As discussed in HBASE-21217, in branch-2.0 and branch-2.1, we should use
> CompatRemoteProcedureResolver instead of ExecuteProceduresRemoteCall to
> dispatch region open/close requests to RS. Since ExecuteProceduresRemoteCall
> will group all the open/close operations in one call and execute them
> sequentially on the target RS. If one operation fails, all the operation will
> be marked as failure. Actually, some of the operations(like open region) is
> already executing in the open region handler thread. But master thinks these
> operations fails and reassign the regions to another RS. So when the previous
> RS report to the master that the region is online, master will kill the RS
> since it already assign the region to another RS.
> For branch-2.2+, HBASE-21217 will fix this issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21387) Race condition in snapshot cache refreshing leads to loss of snapshot files
[
https://issues.apache.org/jira/browse/HBASE-21387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21387:
---
Labels: snapshot (was: )
> Race condition in snapshot cache refreshing leads to loss of snapshot files
> ---
>
> Key: HBASE-21387
> URL: https://issues.apache.org/jira/browse/HBASE-21387
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Labels: snapshot
> Attachments: 21387.v1.txt
>
>
> During recent report from customer where ExportSnapshot failed:
> {code}
> 2018-10-09 18:54:32,559 ERROR [VerifySnapshot-pool1-t2]
> snapshot.SnapshotReferenceUtil: Can't find hfile:
> 44f6c3c646e84de6a63fe30da4fcb3aa in the real
> (hdfs://in.com:8020/apps/hbase/data/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> or archive
> (hdfs://in.com:8020/apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa)
> directory for the primary table.
> {code}
> We found the following in log:
> {code}
> 2018-10-09 18:54:23,675 DEBUG
> [00:16000.activeMasterManager-HFileCleaner.large-1539035367427]
> cleaner.HFileCleaner: Removing:
> hdfs:///apps/hbase/data/archive/data/.../a/44f6c3c646e84de6a63fe30da4fcb3aa
> from archive
> {code}
> The root cause is race condition surrounding SnapshotFileCache#refreshCache().
> There are two callers of refreshCache: one from RefreshCacheTask#run and the
> other from SnapshotHFileCleaner.
> Let's look at the code of refreshCache:
> {code}
> // if the snapshot directory wasn't modified since we last check, we are
> done
> if (dirStatus.getModificationTime() <= this.lastModifiedTime) return;
> // 1. update the modified time
> this.lastModifiedTime = dirStatus.getModificationTime();
> // 2.clear the cache
> this.cache.clear();
> {code}
> Suppose the RefreshCacheTask runs past the if check and sets
> this.lastModifiedTime
> The cleaner executes refreshCache and returns immediately since
> this.lastModifiedTime matches the modification time of the directory.
> Now RefreshCacheTask clears the cache. By the time the cleaner performs cache
> lookup, the cache is empty.
> Therefore cleaner puts the file into unReferencedFiles - leading to data loss.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670238#comment-16670238 ] Josh Elser commented on HBASE-21246: bq. That sounds like a sensible approach (including diagrams bit.. ). Reid Chan doing it doesn't seem right though. Talked to Ted offline. He's working on diagrams today (but thank you, Reid, for the offer). > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.20.txt, 21246.21.txt, > 21246.23.txt, 21246.HBASE-20952.001.patch, 21246.HBASE-20952.002.patch, > 21246.HBASE-20952.004.patch, 21246.HBASE-20952.005.patch, > 21246.HBASE-20952.007.patch, 21246.HBASE-20952.008.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670099#comment-16670099
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #34 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/34/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[ https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670071#comment-16670071 ] Anoop Sam John commented on HBASE-21388: +1 > No need to instantiate MemStoreLAB for master which not carry table > --- > > Key: HBASE-21388 > URL: https://issues.apache.org/jira/browse/HBASE-21388 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21388.master.001.patch, > HBASE-21388.master.002.patch, HBASE-21388.master.003.patch, > HBASE-21388.master.004.patch > > > We found this log in our master. > 2018-10-26,10:00:00,449 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating data > MemStoreChunkPool with chunk size 2 MB, max count 737, initial count 0 > 2018-10-26,10:00:00,452 INFO > [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster] > org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating index > MemStoreChunkPool with chunk size 204.80 KB, max count 819, initial count 0 > > Same with HBASE-21290, we don't need to instantiate MemStore for master which > not carry table. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21401) Sanity check in BaseDecoder#parseCell
[
https://issues.apache.org/jira/browse/HBASE-21401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670038#comment-16670038
]
Hadoop QA commented on HBASE-21401:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s{color}
| {color:red} hbase-common generated 2 new + 42 unchanged - 0 fixed = 44 total
(was 42) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
25s{color} | {color:red} hbase-common: The patch generated 21 new + 148
unchanged - 1 fixed = 169 total (was 149) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hbase-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 5s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21401 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946347/HBASE-21401.v4.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b1d2cb86b63a 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24
13:14:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 02f5f171f5 |
| maven | version: Apache Maven 3.5.4
[jira] [Commented] (HBASE-21374) Backport HBASE-21342 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670032#comment-16670032 ] mazhenlin commented on HBASE-21374: --- Not sure why the test fails , it succeeded in my local run. > Backport HBASE-21342 to branch-1 > > > Key: HBASE-21374 > URL: https://issues.apache.org/jira/browse/HBASE-21374 > Project: HBase > Issue Type: Task >Reporter: Mike Drob >Assignee: mazhenlin >Priority: Major > Attachments: HBASE-21374.branch-1.001.patch, > HBASE-21374.branch-1.002.patch, HBASE-21374.branch-1.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670026#comment-16670026
]
Duo Zhang commented on HBASE-21255:
---
Seems all the recent pre commit builds are broken...
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name and a permission in one of
> [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21255:
--
Comment: was deleted
(was: Seems all the recent pre commit builds are broken...)
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name and a permission in one of
> [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21314) The implementation of BitSetNode is not efficient
[
https://issues.apache.org/jira/browse/HBASE-21314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670021#comment-16670021
]
Hadoop QA commented on HBASE-21314:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s{color}
| {color:red} hbase-procedure in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21314 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946344/HBASE-21314.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ba7bd77b457f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 02f5f171f5 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14910/artifact/patchprocess/patch-unit-hbase-procedure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14910/testReport/ |
| Max. process+thread count | 87 (vs. ulimit of 1) |
| modules | C: hbase-procedure U: hbase-procedure |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14910/console |
| Powered by | Apache
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670016#comment-16670016
]
Duo Zhang commented on HBASE-21255:
---
Seems all the recent pre commit builds are broken...
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name and a permission in one of
> [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669979#comment-16669979
]
Reid Chan commented on HBASE-21255:
---
All modified UTs passed on local, and ShadedAccessControlUtil's checkstyles can
be ignored IMO.
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name and a permission in one of
> [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669973#comment-16669973
]
Reid Chan commented on HBASE-21255:
---
ping [~busbey]
{code}
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
{code}
Do you know what's going on? or something wrong with my patch? Thanks in
advance.
> [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
>
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21225.master.001.patch,
> HBASE-21225.master.002.patch, HBASE-21255.master.003.patch,
> HBASE-21255.master.004.patch
>
>
> A TODO in {{TablePermission.java}}
> {code:java}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
> Change Notes:
> * Divide origin TablePermission into three classes GlobalPermission,
> NamespacePermission, TablePermission
> * New UserPermission consists of a user name and a permission in one of
> [Global, Namespace, Table]Permission.
> * Rename TableAuthManager to AuthManager(it is IA.P), and rename some
> methods for readability.
> * Make PermissionCache thread safe, and the ListMultiMap is changed to Set.
> * User cache and group cache in AuthManager is combined together.
> * Wire proto is kept, BC should be under guarantee.
> * Fix HBASE-21390.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21255) [acl] Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669955#comment-16669955
]
Hadoop QA commented on HBASE-21255:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} The patch passed checkstyle in hbase-common {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} hbase-client: The patch generated 6 new + 79 unchanged
- 30 fixed = 85 total (was 109) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} hbase-server: The patch generated 0 new + 101
unchanged - 53 fixed = 101 total (was 154) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch passed checkstyle in hbase-rsgroup {color}
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
52s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color}
| {color:red} hbase-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 11s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m
[jira] [Commented] (HBASE-21351) The force update thread may have race with PE worker when the procedure is rolling back
[
https://issues.apache.org/jira/browse/HBASE-21351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669935#comment-16669935
]
Hadoop QA commented on HBASE-21351:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
5s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 57s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hbase-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s{color}
| {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 44s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21351 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946318/HBASE-21351-v1.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b6111d7d634e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 02f5f171f5 |
| maven | version: Apache
[jira] [Commented] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[
https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669916#comment-16669916
]
Hadoop QA commented on HBASE-21388:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
59s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 26s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 11s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21388 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946320/HBASE-21388.master.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d96255829abc 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24
13:14:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 02f5f171f5 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14907/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14907/testReport/ |
| Max. process+thread count | 99 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14907/console |
| Powered by | Apache
[jira] [Comment Edited] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669909#comment-16669909 ] Allan Yang edited comment on HBASE-21413 at 10/31/18 10:40 AM: --- Maybe similar with HBASE-14223 https://issues.apache.org/jira/browse/HBASE-14223?focusedCommentId=16640533=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16640533 was (Author: allan163): Maybe similar with HBASE-14223 > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669909#comment-16669909 ] Allan Yang commented on HBASE-21413: Maybe similar with HBASE-14223 > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21407) Resolve NPE in backup Master UI
[
https://issues.apache.org/jira/browse/HBASE-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669888#comment-16669888
]
Hadoop QA commented on HBASE-21407:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 34s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21407 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946324/hbase-21407.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit |
| uname | Linux 9f4d054776d9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 02f5f171f5 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14905/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14905/testReport/ |
| Max. process+thread count | 99 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14905/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Resolve NPE in backup Master UI
>
>
> Key: HBASE-21407
> URL: https://issues.apache.org/jira/browse/HBASE-21407
> Project: HBase
> Issue Type: Bug
> Components: UI
>Affects Versions: 3.0.0, 2.1.0, 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 2.2.0
>
> Attachments: hbase-21407.master.001.patch,
> hbase-21407.master.001.patch
>
>
> Since some pages of our UI are using jsp instead of jamon, the fix of
> HBASE-18263 is not enough. Added the fix of HBASE-18263 to the header.jsp.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669877#comment-16669877 ] Jingyun Tian commented on HBASE-21413: -- screen shot of this problem. !Screenshot from 2018-10-31 18-11-02.png! !Screenshot from 2018-10-31 18-11-11.png! > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21413: - Description: After I restart whole cluster, there is a splitting directory still exists on hdfs. Then I found there is only an empty meta wal file in it. I'll dig into this later. ! was:After I restart whole cluster, there is a splitting directory still exists on hdfs. Then I found there is only an empty meta wal file in it. I'll dig into this later. > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. > ! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21413: - Description: After I restart whole cluster, there is a splitting directory still exists on hdfs. Then I found there is only an empty meta wal file in it. I'll dig into this later. (was: After I restart whole cluster, there is a splitting directory still exists on hdfs. Then I found there is only an empty meta wal file in it. I'll dig into this later. !) > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
Jingyun Tian created HBASE-21413: Summary: Empty meta log doesn't get split when restart whole cluster Key: HBASE-21413 URL: https://issues.apache.org/jira/browse/HBASE-21413 Project: HBase Issue Type: Improvement Reporter: Jingyun Tian Assignee: Jingyun Tian Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from 2018-10-31 18-11-11.png After I restart whole cluster, there is a splitting directory still exists on hdfs. Then I found there is only an empty meta wal file in it. I'll dig into this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingyun Tian updated HBASE-21413: - Attachment: Screenshot from 2018-10-31 18-11-02.png Screenshot from 2018-10-31 18-11-11.png > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: Screenshot from 2018-10-31 18-11-02.png, Screenshot from > 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21388) No need to instantiate MemStoreLAB for master which not carry table
[
https://issues.apache.org/jira/browse/HBASE-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669849#comment-16669849
]
Guanghao Zhang commented on HBASE-21388:
[~anoop.hbase] Any more comments? And copy the comments from ReviewBoard.
{quote}bq. Now tables on master is broken. See HBASE-19831 and HBASE-19785. So
we can do these jobs in the future. 1. Make sure that isTablesOnMaster is false
and isSystemTablesOnlyOnMaster is true can't work. 2. introduce a special
memstore/blockcache(which is smaller than RS) config for
SystemTablesOnlyOnMaster.
{quote}
> No need to instantiate MemStoreLAB for master which not carry table
> ---
>
> Key: HBASE-21388
> URL: https://issues.apache.org/jira/browse/HBASE-21388
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.2.0, 2.1.1, 2.0.2
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Attachments: HBASE-21388.master.001.patch,
> HBASE-21388.master.002.patch, HBASE-21388.master.003.patch,
> HBASE-21388.master.004.patch
>
>
> We found this log in our master.
> 2018-10-26,10:00:00,449 INFO
> [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster]
> org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating data
> MemStoreChunkPool with chunk size 2 MB, max count 737, initial count 0
> 2018-10-26,10:00:00,452 INFO
> [master/c4-hadoop-tst-ct16:42900:becomeActiveMaster]
> org.apache.hadoop.hbase.regionserver.ChunkCreator: Allocating index
> MemStoreChunkPool with chunk size 204.80 KB, max count 819, initial count 0
>
> Same with HBASE-21290, we don't need to instantiate MemStore for master which
> not carry table.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
