[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457622#comment-15457622
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 6s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 46 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 33s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 45s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
25s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 8s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 8s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s {color}
| {color:red} hbase-server-jdk1.7.0_111 with JDK v1.7.0_111 generated 2 new + 4
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 46s {color}
| {color:red} root-jdk1.7.0_111 with JDK v1.7.0_111 generated 2 new + 30
unchanged - 2 fixed = 32 total (was 32) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 543 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 15s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457605#comment-15457605
]
Hudson commented on HBASE-16375:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #6 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/6/])
HBASE-16375 Mapreduce mini cluster using HBaseTestingUtility not setting
(apurtell: rev 00c4bf6b011183046df2452656b9a4bb437972f6)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457604#comment-15457604 ] Hudson commented on HBASE-16547: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #6 (See [https://builds.apache.org/job/HBase-1.3-JDK8/6/]) HBASE-16547 hbase-archetype-builder shell scripts assume bash is (apurtell: rev 9657cd2609365b15783562740d75afd56b65c166) * (edit) hbase-archetypes/hbase-archetype-builder/createArchetypes.sh * (edit) hbase-archetypes/hbase-archetype-builder/installArchetypes.sh > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457548#comment-15457548 ] Hudson commented on HBASE-16101: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #18 (See [https://builds.apache.org/job/HBase-1.2-JDK8/18/]) HBASE-16101 Tool to microbenchmark procedure WAL performance. (appy: rev e2e55481f64b519823644b86d575a1c01300bcf3) * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/util/AbstractHBaseTool.java * (add) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALLoaderPerformanceEvaluation.java * (add) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALPerformanceEvaluation.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java * (edit) hbase-assembly/pom.xml > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457549#comment-15457549
]
Hudson commented on HBASE-16375:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #18 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/18/])
HBASE-16375 Mapreduce mini cluster using HBaseTestingUtility not setting
(apurtell: rev 074fbcfc5ff79cf5617839089e41d886e8c514cf)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15513) hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default
[ https://issues.apache.org/jira/browse/HBASE-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457499#comment-15457499 ] ramkrishna.s.vasudevan commented on HBASE-15513: The reason why chunk pool is better with G1GC is as Anoop stated but that is for lower heap sizes. For higher heap sizes may be it is not needed to be ON by default? So am not very sure whether we should always enable this chunkpool. When we use offheap MSLAB then yes, it has to be ON, so that we avoid offheap memory fragmentation. > hbase.hregion.memstore.chunkpool.maxsize is 0.0 by default > -- > > Key: HBASE-15513 > URL: https://issues.apache.org/jira/browse/HBASE-15513 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-15513-v1.patch > > > That results in excessive MemStoreLAB chunk allocations because we can not > reuse them. Not sure, why it has been disabled, by default. May be the code > has not been tested well? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457490#comment-15457490
]
Hadoop QA commented on HBASE-16538:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 0s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 8s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-02 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826762/HBASE-16538.master.002.patch
|
| JIRA Issue | HBASE-16538 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
[jira] [Updated] (HBASE-16507) Procedure v2 - Force DDL operation to always roll forward
[ https://issues.apache.org/jira/browse/HBASE-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16507: Resolution: Fixed Status: Resolved (was: Patch Available) > Procedure v2 - Force DDL operation to always roll forward > - > > Key: HBASE-16507 > URL: https://issues.apache.org/jira/browse/HBASE-16507 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16507-v0.patch, HBASE-16507-v1.patch > > > Having rollback for DDLs was a bad idea. > and it turns out to be an unexpected behavior for the user. > DDLs only have transient errors (e.g. zk, hdfs, meta down) > if we abort/rollback on a transient failure the user will get a failure, > and it is not clear why the user needs to retry the command when the system > can do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457451#comment-15457451 ] Hudson commented on HBASE-16101: FAILURE: Integrated in Jenkins build HBase-1.4 #389 (See [https://builds.apache.org/job/HBase-1.4/389/]) HBASE-16101 addendum. Fixes runtime ClassCastException of (appy: rev 4147dcb81a8e2dab6b707b8e132b6c325b8cca0a) * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALPerformanceEvaluation.java > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457423#comment-15457423 ] Hudson commented on HBASE-16547: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #7 (See [https://builds.apache.org/job/HBase-1.3-JDK7/7/]) HBASE-16547 hbase-archetype-builder shell scripts assume bash is (apurtell: rev 9657cd2609365b15783562740d75afd56b65c166) * (edit) hbase-archetypes/hbase-archetype-builder/createArchetypes.sh * (edit) hbase-archetypes/hbase-archetype-builder/installArchetypes.sh > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457424#comment-15457424
]
Hudson commented on HBASE-16375:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #7 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/7/])
HBASE-16375 Mapreduce mini cluster using HBaseTestingUtility not setting
(apurtell: rev 00c4bf6b011183046df2452656b9a4bb437972f6)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457414#comment-15457414 ] Mikhail Antonov commented on HBASE-16101: - [~appy] as it's addendum to already committed stuff and test code anyway, +1 - thanks for the ping! > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-16538:
-
Attachment: HBASE-16538.master.002.patch
> Version mismatch in HBaseConfiguration.checkDefaultsVersion
> ---
>
> Key: HBASE-16538
> URL: https://issues.apache.org/jira/browse/HBASE-16538
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
> Labels: configuration, test-failure
> Attachments: HBASE-16538.master.001.patch,
> HBASE-16538.master.002.patch
>
>
> {noformat}
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures
> testYieldEachExecutionStep(org.apache.hadoop.hbase.procedure2.TestYieldProcedures)
> Time elapsed: 0.255 sec <<< ERROR!
> java.lang.RuntimeException: hbase-default.xml file seems to be for an older
> version of HBase (2.0.0-SNAPSHOT), this version is Unknown
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:83)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at
> org.apache.hadoop.hbase.HBaseCommonTestingUtility.(HBaseCommonTestingUtility.java:46)
> at
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures.setUp(TestYieldProcedures.java:63)
> {noformat}
> (Exact test is not important)
> Reference run:
> https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1515/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16524) Clean procedure wal periodically instead of on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457397#comment-15457397 ] Matteo Bertozzi edited comment on HBASE-16524 at 9/2/16 3:42 AM: - before adding conditions to reduce the number of times we call removeInactiveLogs(), let's try to improve the algorithm. executing every roll does not provide the result we want. e.g. WAL-1: [1, 2], Current: [] as soon we update [1, 2] we can remove WAL-1 e.g. WAL-1: [1], WAL-2: [2], Current [] as soon we update [1, 2] we can remove WAL-1 and WAL-2 was (Author: mbertozzi): before adding conditions to reduce the number of times we call removeInactiveLogs(), let's try to improve the algorithm. executing every roll does not provide the result we want. e.g. WAL-1: [1, 2] Current: [] as soon we update [1, 2] we can remove WAL-1 e.g. WAL-1: [1], WAL-2: [2] as soon we update [1, 2] we can remove WAL-1 and WAL-2 > Clean procedure wal periodically instead of on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16524.master.001.patch, flame1.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16524) Clean procedure wal periodically instead of on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457397#comment-15457397 ] Matteo Bertozzi commented on HBASE-16524: - before adding conditions to reduce the number of times we call removeInactiveLogs(), let's try to improve the algorithm. executing every roll does not provide the result we want. e.g. WAL-1: [1, 2] Current: [] as soon we update [1, 2] we can remove WAL-1 e.g. WAL-1: [1], WAL-2: [2] as soon we update [1, 2] we can remove WAL-1 and WAL-2 > Clean procedure wal periodically instead of on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16524.master.001.patch, flame1.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[
https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457388#comment-15457388
]
Phil Yang commented on HBASE-16505:
---
In the first step, we may not use new thread. We may just use the interface
like this:
{code}
asyncRegion.get(context, get);
context.getResult()
{code}
The first line is still blocking, and second line will return the result
immediately. The only goal is pass a context with deadline for HBASE-16492.
The reason to change the interface to async( instead of add a context first and
then change it to async) is to prevent changing it another time. The interface
is important so we may reduce the time of changing.
In the second step, we can make it truly non-blocking, we not only do the get
logic in another thread pool, but also use another thread pool to getResult and
send response to client.
> Add AsyncRegion interface to pass deadline and support async operations
> ---
>
> Key: HBASE-16505
> URL: https://issues.apache.org/jira/browse/HBASE-16505
> Project: HBase
> Issue Type: Sub-task
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16505-v1.patch
>
>
> If we want to know the correct setting of timeout in read/write path, we need
> add a new parameter in operation-methods of Region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16550) Procedure v2 - Add AM compatibility for 2.x Master and 1.x RSs
Matteo Bertozzi created HBASE-16550: --- Summary: Procedure v2 - Add AM compatibility for 2.x Master and 1.x RSs Key: HBASE-16550 URL: https://issues.apache.org/jira/browse/HBASE-16550 Project: HBase Issue Type: Sub-task Components: proc-v2, Region Assignment Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 Core AM HBASE-14614 relies on the RS to be using zkless assignment. Add support for the old a plain non zkless AM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16549) Procedure v2 - Add new AM metrics
Matteo Bertozzi created HBASE-16549: --- Summary: Procedure v2 - Add new AM metrics Key: HBASE-16549 URL: https://issues.apache.org/jira/browse/HBASE-16549 Project: HBase Issue Type: Sub-task Components: proc-v2, Region Assignment Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 With the new AM we can add a bunch of metrics - assign/unassign time - server crash time - grouping related metrics? (how many batch we do, and similar?) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14615) Procedure V2: Implement single region assignment procedure from Master-side
[ https://issues.apache.org/jira/browse/HBASE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14615: Resolution: Invalid Status: Resolved (was: Patch Available) Closing since this is part of HBASE-14614 > Procedure V2: Implement single region assignment procedure from Master-side > --- > > Key: HBASE-14615 > URL: https://issues.apache.org/jira/browse/HBASE-14615 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > Attachments: HBASE-14615.v0-master.patch > > > this JIRA implements assignment procedure from Master-side. > - send open/close/offline RPC to RS for assignment (involves table and server > queues) > - wait for response > - listen to dispatcher and wait for completion (timeout, success, fail) > - if procedure fails due to dead server, deal with it (let SSH handle or > retry with a different RS) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16548) Procedure v2 - Add handling of split/merge region transition to the new AM
Matteo Bertozzi created HBASE-16548: --- Summary: Procedure v2 - Add handling of split/merge region transition to the new AM Key: HBASE-16548 URL: https://issues.apache.org/jira/browse/HBASE-16548 Project: HBase Issue Type: Sub-task Components: proc-v2, Region Assignment Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 Core Assignment HBASE-14614 does not handle split and merge in reportRegionStateTransition(). Handle the transition request! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457272#comment-15457272 ] Hudson commented on HBASE-16527: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #20 (See [https://builds.apache.org/job/HBase-1.2-JDK7/20/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev 81655cab9600542edbc76ff21bb9f65045b7e648) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16340) ensure no Xerces jars included
[ https://issues.apache.org/jira/browse/HBASE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457271#comment-15457271 ] Hudson commented on HBASE-16340: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #20 (See [https://builds.apache.org/job/HBase-1.2-JDK7/20/]) HBASE-16340 exclude Xerces iplementation jars from coming in (apurtell: rev 0f7bfe8321481df378cee64b9949566824ff6421) * (edit) pom.xml > ensure no Xerces jars included > -- > > Key: HBASE-16340 > URL: https://issues.apache.org/jira/browse/HBASE-16340 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16340.1.patch, HBASE-16340.2.patch > > > When we moved our pom to Hadoop 2.7 we picked up a transitive Xerces > implementation. We should exclude this to ensure we don't get a conflict with > the implementation that ships with the jvm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457273#comment-15457273
]
Hudson commented on HBASE-16375:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #20 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/20/])
HBASE-16375 Mapreduce mini cluster using HBaseTestingUtility not setting
(apurtell: rev 074fbcfc5ff79cf5617839089e41d886e8c514cf)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14617) Procedure V2: Update ServerCrashProcedure to interact with assignment procedures
[ https://issues.apache.org/jira/browse/HBASE-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi resolved HBASE-14617. - Resolution: Invalid Part of HBASE-14616 > Procedure V2: Update ServerCrashProcedure to interact with assignment > procedures > > > Key: HBASE-14617 > URL: https://issues.apache.org/jira/browse/HBASE-14617 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > > this JIRA tracks the update of ServerCrashProcedure to interact with > assignment procedures. This is very critical (and most tricky part of) work > that deals with dead region server when assignment is happening. > - remove region server queue when the region server is dead > - notify assignment procedures that are doing assignment operation in the > dead server > - assign regions in dead server to other RS (we have to deal with RIT, deal > with in-progress table DDL) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14616) Procedure v2: Replace the old AM with the new AM
[ https://issues.apache.org/jira/browse/HBASE-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi reassigned HBASE-14616: --- Assignee: Matteo Bertozzi (was: Stephen Yuan Jiang) > Procedure v2: Replace the old AM with the new AM > > > Key: HBASE-14616 > URL: https://issues.apache.org/jira/browse/HBASE-14616 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > > Replace the old AssignmentManager with the new one HBASE-14614. > This will touch most of the DDLs, and HMaster code. ServerCrashProcedure is > also part of this integration change -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14616) Procedure v2: Replace the old AM with the new AM
[ https://issues.apache.org/jira/browse/HBASE-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14616: Description: Replace the old AssignmentManager with the new one HBASE-14614. This will touch most of the DDLs, and HMaster code. ServerCrashProcedure is also part of this integration change was: This JIRA modifies the assignment operations inside table DDLs by calling assignment procedure (HBASE-14615) - Inside table DDL, if any DDL involves region assignment, assignment procedures should be used as child procedure. - Wait for child procedure completes, based on result (timeout, success, failure), decide next action > Procedure v2: Replace the old AM with the new AM > > > Key: HBASE-14616 > URL: https://issues.apache.org/jira/browse/HBASE-14616 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > > Replace the old AssignmentManager with the new one HBASE-14614. > This will touch most of the DDLs, and HMaster code. ServerCrashProcedure is > also part of this integration change -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14616) Procedure v2: Replace the old AM with the new AM
[ https://issues.apache.org/jira/browse/HBASE-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14616: Summary: Procedure v2: Replace the old AM with the new AM (was: Procedure V2: Migrate assignment operation inside table DDLs by calling assignment procedure) > Procedure v2: Replace the old AM with the new AM > > > Key: HBASE-14616 > URL: https://issues.apache.org/jira/browse/HBASE-14616 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0 > > > This JIRA modifies the assignment operations inside table DDLs by calling > assignment procedure (HBASE-14615) > - Inside table DDL, if any DDL involves region assignment, assignment > procedures should be used as child procedure. > - Wait for child procedure completes, based on result (timeout, success, > failure), decide next action -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14614: Description: New AssignmentManager implemented using proc-v2. - AssignProcedure handle assignment operation - UnassignProcedure handle unassign operation - MoveRegionProcedure handle move/balance operation Concurrent Assign operations are batched together and sent to the balancer Concurrent Assign and Unassign operation ready to be sent to the RS are batched together This patch is an intermediate state where we add the new AM as AssignmentManager2() to the master, to be reached by tests. but the new AM will not be integrated with the rest of the system. Only new am unit-tests will exercise the new assigment manager. The integration with the master code is part of HBASE-14616 was: New AssignmentManager implemented using proc-v2. - AssignProcedure handle assignment operation - UnassignProcedure handle unassign operation - MoveRegionProcedure handle move/balance operation Concurrent Assign operations are batched together and sent to the balancer Concurrent Assign and Unassign operation ready to be sent to the RS are batched together This patch is an intermediate state where we add the new AM as AssignmentManager2() to the master, to be reached by tests. but the new AM will not be integrated with the rest of the system. Only new am unit-tests will exercise the new assigment manager. The integration with the master code is part of > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of HBASE-14616 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14614: Description: New AssignmentManager implemented using proc-v2. - AssignProcedure handle assignment operation - UnassignProcedure handle unassign operation - MoveRegionProcedure handle move/balance operation Concurrent Assign operations are batched together and sent to the balancer Concurrent Assign and Unassign operation ready to be sent to the RS are batched together This patch is an intermediate state where we add the new AM as AssignmentManager2() to the master, to be reached by tests. but the new AM will not be integrated with the rest of the system. Only new am unit-tests will exercise the new assigment manager. The integration with the master code is part of was:Within a small interval, there could exists multiple region assignment operations against 1 region server. To improve performance and reduce number of RPCs between MASTER and RS, a dispatch will be implemented to group multiple region assignment operation to 1 RPC (batch). > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > > New AssignmentManager implemented using proc-v2. > - AssignProcedure handle assignment operation > - UnassignProcedure handle unassign operation > - MoveRegionProcedure handle move/balance operation > Concurrent Assign operations are batched together and sent to the balancer > Concurrent Assign and Unassign operation ready to be sent to the RS are > batched together > This patch is an intermediate state where we add the new AM as > AssignmentManager2() to the master, to be reached by tests. but the new AM > will not be integrated with the rest of the system. Only new am unit-tests > will exercise the new assigment manager. The integration with the master code > is part of -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16541) Avoid unnecessary cell copy in Result.compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457257#comment-15457257
]
Anoop Sam John commented on HBASE-16541:
Planning to give a patch?
> Avoid unnecessary cell copy in Result.compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Priority: Trivial
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16541) Avoid unnecessary cell copy in Result.compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-16541:
---
Priority: Major (was: Trivial)
> Avoid unnecessary cell copy in Result.compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
[ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14614: Summary: Procedure v2: Core Assignment Manager (was: Procedure V2: Implement a Dispatch to group multiple assign operation and send batch RPC to 1 RS) > Procedure v2: Core Assignment Manager > - > > Key: HBASE-14614 > URL: https://issues.apache.org/jira/browse/HBASE-14614 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > > Within a small interval, there could exists multiple region assignment > operations against 1 region server. To improve performance and reduce number > of RPCs between MASTER and RS, a dispatch will be implemented to group > multiple region assignment operation to 1 RPC (batch). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457246#comment-15457246
]
Hadoop QA commented on HBASE-16538:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 1s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s
{color} | {color:red} hbase-common generated 6 new + 0 unchanged - 0 fixed = 6
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 48s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 54s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-common |
| | org.apache.hadoop.hbase.Version.date isn't final but should be At
Version.java:be At Version.java:[line 10] |
| | org.apache.hadoop.hbase.Version.revision isn't final but should be At
Version.java:be At
[jira] [Commented] (HBASE-16516) Revisit the implementation of PayloadCarryingRpcController
[
https://issues.apache.org/jira/browse/HBASE-16516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457229#comment-15457229
]
Duo Zhang commented on HBASE-16516:
---
All green, good.
[~stack] Any concerns? I can prepare the final patch of HBASE-16445 after this
patch committed.
Thanks.
> Revisit the implementation of PayloadCarryingRpcController
> --
>
> Key: HBASE-16516
> URL: https://issues.apache.org/jira/browse/HBASE-16516
> Project: HBase
> Issue Type: Sub-task
> Components: rpc
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16516-v1.patch, HBASE-16516-v2.patch,
> HBASE-16516-v3.patch, HBASE-16516-v4.patch, HBASE-16516.patch
>
>
> First, it should be an interface, the current implementation of
> {{DelegatingPayloadCarryingRpcController}} is weird.
> Second, we need to be more careful when dealing with cancel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15165) AsyncProcess can spin wait indefinitly
[
https://issues.apache.org/jira/browse/HBASE-15165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457222#comment-15457222
]
Hadoop QA commented on HBASE-15165:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-02 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826751/HBASE-15165.002.patch
|
| JIRA Issue | HBASE-15165 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5482dd8362cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 5c7fa12 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457189#comment-15457189 ] Hudson commented on HBASE-16101: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1524 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1524/]) HBASE-16101 addendum. Fixes runtime ClassCastException of (appy: rev 5c7fa12ab31d840955454c76d72b799486036313) * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALPerformanceEvaluation.java > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457190#comment-15457190
]
Hudson commented on HBASE-16375:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1524 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1524/])
HBASE-16375 Mapreduce mini cluster using HBaseTestingUtility not setting
(apurtell: rev b839d04d8e2cc3dcadeddf08203591466650ac3e)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457188#comment-15457188 ] Hudson commented on HBASE-16547: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1524 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1524/]) HBASE-16547 hbase-archetype-builder shell scripts assume bash is (apurtell: rev 817660dd11ef9758b953abcc7b671ab305f6c63f) * (edit) hbase-archetypes/hbase-archetype-builder/installArchetypes.sh * (edit) hbase-archetypes/hbase-archetype-builder/createArchetypes.sh > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-16538:
-
Attachment: HBASE-16538.master.001.patch
> Version mismatch in HBaseConfiguration.checkDefaultsVersion
> ---
>
> Key: HBASE-16538
> URL: https://issues.apache.org/jira/browse/HBASE-16538
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
> Labels: configuration, test-failure
> Attachments: HBASE-16538.master.001.patch
>
>
> {noformat}
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures
> testYieldEachExecutionStep(org.apache.hadoop.hbase.procedure2.TestYieldProcedures)
> Time elapsed: 0.255 sec <<< ERROR!
> java.lang.RuntimeException: hbase-default.xml file seems to be for an older
> version of HBase (2.0.0-SNAPSHOT), this version is Unknown
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:83)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at
> org.apache.hadoop.hbase.HBaseCommonTestingUtility.(HBaseCommonTestingUtility.java:46)
> at
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures.setUp(TestYieldProcedures.java:63)
> {noformat}
> (Exact test is not important)
> Reference run:
> https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1515/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-16538:
-
Status: Patch Available (was: Open)
> Version mismatch in HBaseConfiguration.checkDefaultsVersion
> ---
>
> Key: HBASE-16538
> URL: https://issues.apache.org/jira/browse/HBASE-16538
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
> Labels: configuration, test-failure
> Attachments: HBASE-16538.master.001.patch
>
>
> {noformat}
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures
> testYieldEachExecutionStep(org.apache.hadoop.hbase.procedure2.TestYieldProcedures)
> Time elapsed: 0.255 sec <<< ERROR!
> java.lang.RuntimeException: hbase-default.xml file seems to be for an older
> version of HBase (2.0.0-SNAPSHOT), this version is Unknown
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:83)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at
> org.apache.hadoop.hbase.HBaseCommonTestingUtility.(HBaseCommonTestingUtility.java:46)
> at
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures.setUp(TestYieldProcedures.java:63)
> {noformat}
> (Exact test is not important)
> Reference run:
> https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1515/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457167#comment-15457167 ] Hudson commented on HBASE-16527: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #5 (See [https://builds.apache.org/job/HBase-1.3-JDK8/5/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev b41565d21ed1f2f290990ea0a86379910c052425) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15165) AsyncProcess can spin wait indefinitly
[ https://issues.apache.org/jira/browse/HBASE-15165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengyue Li updated HBASE-15165: --- Attachment: HBASE-15165.002.patch Fix merge conflicts > AsyncProcess can spin wait indefinitly > -- > > Key: HBASE-15165 > URL: https://issues.apache.org/jira/browse/HBASE-15165 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Elliott Clark >Assignee: Pengyue Li >Priority: Critical > Attachments: HBASE-15165.001.patch, HBASE-15165.002.patch > > > When the max outstanding requests per region or per server is reached, all > threads trying to send more requests to that server will spin and will spin > forever with no sleep, and no regard for timeouts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457132#comment-15457132 ] Appy commented on HBASE-16101: -- Hey [~mantonov], there's a small addendum (attached) related to this patch. i have already pushed it to other branches. Looking for your green flag to push in 1.3 > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457120#comment-15457120
]
Andrew Purtell commented on HBASE-16538:
bq. Or, we can get rid of annotations, and directly write a class from
[saveVersion.sh|https://github.com/apache/hbase/blob/master/hbase-common/src/saveVersion.sh]
with static members.
This is what ZooKeeper and Hadoop do, IIRC
> Version mismatch in HBaseConfiguration.checkDefaultsVersion
> ---
>
> Key: HBASE-16538
> URL: https://issues.apache.org/jira/browse/HBASE-16538
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
> Labels: configuration, test-failure
>
> {noformat}
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures
> testYieldEachExecutionStep(org.apache.hadoop.hbase.procedure2.TestYieldProcedures)
> Time elapsed: 0.255 sec <<< ERROR!
> java.lang.RuntimeException: hbase-default.xml file seems to be for an older
> version of HBase (2.0.0-SNAPSHOT), this version is Unknown
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:83)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at
> org.apache.hadoop.hbase.HBaseCommonTestingUtility.(HBaseCommonTestingUtility.java:46)
> at
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures.setUp(TestYieldProcedures.java:63)
> {noformat}
> (Exact test is not important)
> Reference run:
> https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1515/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457113#comment-15457113 ] Hudson commented on HBASE-16527: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #17 (See [https://builds.apache.org/job/HBase-1.2-JDK8/17/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev 81655cab9600542edbc76ff21bb9f65045b7e648) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16340) ensure no Xerces jars included
[ https://issues.apache.org/jira/browse/HBASE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457112#comment-15457112 ] Hudson commented on HBASE-16340: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #17 (See [https://builds.apache.org/job/HBase-1.2-JDK8/17/]) HBASE-16340 exclude Xerces iplementation jars from coming in (apurtell: rev 0f7bfe8321481df378cee64b9949566824ff6421) * (edit) pom.xml > ensure no Xerces jars included > -- > > Key: HBASE-16340 > URL: https://issues.apache.org/jira/browse/HBASE-16340 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16340.1.patch, HBASE-16340.2.patch > > > When we moved our pom to Hadoop 2.7 we picked up a transitive Xerces > implementation. We should exclude this to ensure we don't get a conflict with > the implementation that ships with the jvm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457102#comment-15457102
]
Appy commented on HBASE-16538:
--
I have an alternate suggestion, mainly because of the situation we are in here.
Even if we do the temporary fix, and maybe we find the issue (that indeed,
sometimes annotations don't get loaded soon enough), i don't think we can do
much since that's java for us.
Or, we can get rid of annotations, and directly write a class from
[saveVersion.sh|https://github.com/apache/hbase/blob/master/hbase-common/src/saveVersion.sh]
with static members.
it's a simple idea, so it's possible that whoever wrote the annotation thing
tried it out and found that it didn't work. :)
> Version mismatch in HBaseConfiguration.checkDefaultsVersion
> ---
>
> Key: HBASE-16538
> URL: https://issues.apache.org/jira/browse/HBASE-16538
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
> Labels: configuration, test-failure
>
> {noformat}
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures
> testYieldEachExecutionStep(org.apache.hadoop.hbase.procedure2.TestYieldProcedures)
> Time elapsed: 0.255 sec <<< ERROR!
> java.lang.RuntimeException: hbase-default.xml file seems to be for an older
> version of HBase (2.0.0-SNAPSHOT), this version is Unknown
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:83)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at
> org.apache.hadoop.hbase.HBaseCommonTestingUtility.(HBaseCommonTestingUtility.java:46)
> at
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures.setUp(TestYieldProcedures.java:63)
> {noformat}
> (Exact test is not important)
> Reference run:
> https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1515/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457068#comment-15457068 ] Ted Yu commented on HBASE-15449: Patch is on review board: https://reviews.apache.org/r/51586/diff/1#index_header > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457064#comment-15457064 ] Ted Yu commented on HBASE-15449: https://issues.apache.org/jira/secure/attachment/12826716/15449.v8.txt should apply. I checked RestoreServerUtil.java in the above patch just now where the conflict is resolved. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16101: - Attachment: HBASE-16101-addendum.patch > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16101) Procedure v2 - Perf Tool for WAL
[ https://issues.apache.org/jira/browse/HBASE-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy reopened HBASE-16101: -- > Procedure v2 - Perf Tool for WAL > > > Key: HBASE-16101 > URL: https://issues.apache.org/jira/browse/HBASE-16101 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, tooling >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.3 > > Attachments: HBASE-16101-addendum.patch, > HBASE-16101.master.001.patch, HBASE-16101.master.002.patch, > HBASE-16101.master.003.patch, HBASE-16101.master.004.patch, > HBASE-16101.master.005.patch, HBASE-16101.master.006.patch, > HBASE-16101.master.007.patch, HBASE-16101.master.008.patch, > HBASE-16101.master.009.patch, HBASE-16101.master.010.patch > > > Add upstream tools to measure procedure perf > There are 2 main thing to measure: > - Procedure Loading > - Procedure Writing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457049#comment-15457049 ] Vladimir Rodionov commented on HBASE-15449: --- I do not see patch, [~tedyu] Have you attached it? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[
https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457041#comment-15457041
]
Vladimir Rodionov commented on HBASE-15449:
---
[~tedyu], still does not apply
{code}
HW10692:hbase vrodionov$ patch -p1 < 15449.v8.txt
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/RestoreClientImpl.java
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/BackupServerUtil.java
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/RestoreServerUtil.java
Hunk #3 FAILED at 59.
Hunk #4 succeeded at 140 (offset 3 lines).
Hunk #5 succeeded at 200 (offset 3 lines).
Hunk #6 succeeded at 236 (offset 3 lines).
Hunk #7 succeeded at 344 (offset 3 lines).
Hunk #8 succeeded at 567 (offset 3 lines).
Hunk #9 succeeded at 667 (offset 3 lines).
Hunk #10 succeeded at 685 (offset 3 lines).
Hunk #11 succeeded at 706 with fuzz 1 (offset 11 lines).
1 out of 11 hunks FAILED -- saving rejects to file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/RestoreServerUtil.java.rej
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
patching file
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java
patching file
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackup.java
{code}
> HBase Backup Phase 3: Support physical table layout change
> ---
>
> Key: HBASE-15449
> URL: https://issues.apache.org/jira/browse/HBASE-15449
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Ted Yu
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt,
> 15449.v7.txt, 15449.v8.txt
>
>
> Table operation such as add column family, delete column family, truncate ,
> delete table may result in subsequent backup restore failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457017#comment-15457017
]
Hudson commented on HBASE-16375:
FAILURE: Integrated in Jenkins build HBase-1.4 #388 (See
[https://builds.apache.org/job/HBase-1.4/388/])
HBASE-16375 Mapreduce mini cluster using HBaseTestingUtility not setting
(apurtell: rev 84b24314727e91e693435d050b0a36e5df4bb1a8)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestHBaseTestingUtility.java
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457015#comment-15457015 ] Hudson commented on HBASE-16547: FAILURE: Integrated in Jenkins build HBase-1.4 #388 (See [https://builds.apache.org/job/HBase-1.4/388/]) HBASE-16547 hbase-archetype-builder shell scripts assume bash is (apurtell: rev 77a6c6ce346a1345548d21642bcb89793855e1d2) * (edit) hbase-archetypes/hbase-archetype-builder/installArchetypes.sh * (edit) hbase-archetypes/hbase-archetype-builder/createArchetypes.sh > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457007#comment-15457007 ] Andrew Purtell commented on HBASE-16547: bq. Perhaps /usr/bin/env bash should be the shebang we enforce in future shell scripts? I considered that, and think only where we need bash because we've scripted something that requires one of its features > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Attachment: 14123-master.v16.txt > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v2.txt, 14123-master.v3.txt, > 14123-master.v5.txt, 14123-master.v6.txt, 14123-master.v7.txt, > 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[
https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457002#comment-15457002
]
Dima Spivak commented on HBASE-16547:
-
Perhaps {{/usr/bin/env bash}} should be the shebang we enforce in future shell
scripts? Looking at {{installArchetypes.sh}}, I can't help but think that maybe
it's doing things that the {{pom.xml}} in that module should be handling, but
that's a different story.
> hbase-archetype-builder shell scripts assume bash is installed in /bin
> --
>
> Key: HBASE-16547
> URL: https://issues.apache.org/jira/browse/HBASE-16547
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16547.patch
>
>
> There's no guarantee UNIX systems will have bash installed in /bin. HBase
> builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use
> any bash features so let's specify '/bin/sh' as interpreter instead.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16311) Audit log for delete snapshot operation is missing in case of snapshot owner deleting the same
[
https://issues.apache.org/jira/browse/HBASE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457000#comment-15457000
]
Jerry He commented on HBASE-16311:
--
+1
> Audit log for delete snapshot operation is missing in case of snapshot owner
> deleting the same
> --
>
> Key: HBASE-16311
> URL: https://issues.apache.org/jira/browse/HBASE-16311
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Abhishek Kumar
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16311-V1.patch, HBASE-16311-V2.patch,
> HBASE-16311-V3.patch, HBASE-16311-V4.patch
>
>
> 1. Audit log seems to be left as a TODO task in AccessController.java:
> {code}
> @Override
> public void preDeleteSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot) throws IOException {
> if (SnapshotDescriptionUtils.isSnapshotOwner(snapshot, getActiveUser())) {
> // Snapshot owner is allowed to delete the snapshot
> // TODO: We are not logging this for audit
> } else {
> requirePermission("deleteSnapshot", Action.ADMIN);
> }
> }
> {code}
> 2. Also, snapshot name is not getting logged in the audit logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16445) Refactor and reimplement RpcClient
[ https://issues.apache.org/jira/browse/HBASE-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456996#comment-15456996 ] Duo Zhang commented on HBASE-16445: --- Yes, the patch here is a first try that contains everything. I will prepare a new patch after the other sub tasks committed. > Refactor and reimplement RpcClient > -- > > Key: HBASE-16445 > URL: https://issues.apache.org/jira/browse/HBASE-16445 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16445-v1.patch, HBASE-16445.patch > > > There are lots of common logics between RpcClientImpl and AsyncRpcClient. We > should have much less code comparing to the current implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456986#comment-15456986 ] Hudson commented on HBASE-16527: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1777 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1777/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev bd61885ba133999f4bd76fff5ce9950a4d3d0010) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16340) ensure no Xerces jars included
[ https://issues.apache.org/jira/browse/HBASE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456985#comment-15456985 ] Hudson commented on HBASE-16340: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1777 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1777/]) HBASE-16340 exclude Xerces iplementation jars from coming in (apurtell: rev 69a64d2e347746fbf37e2b7e16cb8573fab7175f) * (edit) pom.xml > ensure no Xerces jars included > -- > > Key: HBASE-16340 > URL: https://issues.apache.org/jira/browse/HBASE-16340 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16340.1.patch, HBASE-16340.2.patch > > > When we moved our pom to Hadoop 2.7 we picked up a transitive Xerces > implementation. We should exclude this to ensure we don't get a conflict with > the implementation that ships with the jvm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456974#comment-15456974 ] Hudson commented on HBASE-16527: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #6 (See [https://builds.apache.org/job/HBase-1.3-JDK7/6/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev b41565d21ed1f2f290990ea0a86379910c052425) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456955#comment-15456955
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 46 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 56s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 31s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
14s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 53s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 35s {color}
| {color:red} hbase-server-jdk1.7.0_111 with JDK v1.7.0_111 generated 2 new + 4
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 38s {color}
| {color:red} root-jdk1.7.0_111 with JDK v1.7.0_111 generated 2 new + 30
unchanged - 2 fixed = 32 total (was 32) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
5s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 543 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 14s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456937#comment-15456937 ] Hudson commented on HBASE-16527: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1862 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1862/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev bd61885ba133999f4bd76fff5ce9950a4d3d0010) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16445) Refactor and reimplement RpcClient
[ https://issues.apache.org/jira/browse/HBASE-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456934#comment-15456934 ] Enis Soztutar commented on HBASE-16445: --- The patch contains changes from other subtasks as well? > Refactor and reimplement RpcClient > -- > > Key: HBASE-16445 > URL: https://issues.apache.org/jira/browse/HBASE-16445 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16445-v1.patch, HBASE-16445.patch > > > There are lots of common logics between RpcClientImpl and AsyncRpcClient. We > should have much less code comparing to the current implementations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-16547. Resolution: Fixed Pushed trivial bugfix > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16375:
---
Attachment: HBASE-16375.patch
Attaching what I committed as 'HBASE-16375.patch'
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch, HBASE-16375.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-16375:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16524) Clean procedure wal periodically instead of on every sync
[ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16524: - Attachment: flame1.svg > Clean procedure wal periodically instead of on every sync > - > > Key: HBASE-16524 > URL: https://issues.apache.org/jira/browse/HBASE-16524 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16524.master.001.patch, flame1.svg > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16524) Clean procedure wal periodically instead of on every sync
[
https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456914#comment-15456914
]
Appy commented on HBASE-16524:
--
Attaching flame graph form a run of {{./bin/hbase
org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALPerformanceEvaluation
-procs 50}} (captured 15sec).
As expected, bottleneck is subtraction of two bitmaps.
The lowest hanging fruit here is to not call removeInactiveLogs() on every sync.
Also, instead of waiting for a predetermined (and randomly) chosen time, it'll
be better to call the cleaner once every wal roll. Only the written wals (not
the active one) are used to decided the obsolete ones, therefore, calling
removeInactiveLogs() repeatedly until the active log is rolled won't mark any
new logs for removal.
> Clean procedure wal periodically instead of on every sync
> -
>
> Key: HBASE-16524
> URL: https://issues.apache.org/jira/browse/HBASE-16524
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
>Priority: Minor
> Attachments: HBASE-16524.master.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16547: --- Assignee: Andrew Purtell Fix Version/s: 1.4.0 1.3.0 2.0.0 > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16547: --- Attachment: HBASE-16547.patch > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Priority: Trivial > Attachments: HBASE-16547.patch > > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456871#comment-15456871
]
Andrew Purtell commented on HBASE-16375:
I'm going to modify this patch slightly. On branch-1 and below I'm seeing
failures due to missing jsr305 annotations for Configuration due to the new
test. (OpenJDK 7u111) If I change the test to use HBaseTestingUtility's
getConfiguration method to access the configuration object there's no problem.
Interestingly this does not manifest on master branch. I will come back to this
later in the context of the change that banned jsr305.
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
[ https://issues.apache.org/jira/browse/HBASE-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16547: --- Priority: Trivial (was: Major) > hbase-archetype-builder shell scripts assume bash is installed in /bin > -- > > Key: HBASE-16547 > URL: https://issues.apache.org/jira/browse/HBASE-16547 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Priority: Trivial > > There's no guarantee UNIX systems will have bash installed in /bin. HBase > builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use > any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16547) hbase-archetype-builder shell scripts assume bash is installed in /bin
Andrew Purtell created HBASE-16547: -- Summary: hbase-archetype-builder shell scripts assume bash is installed in /bin Key: HBASE-16547 URL: https://issues.apache.org/jira/browse/HBASE-16547 Project: HBase Issue Type: Bug Reporter: Andrew Purtell There's no guarantee UNIX systems will have bash installed in /bin. HBase builds fail for me on FreeBSD. The hbase-archetype-builder scripts do not use any bash features so let's specify '/bin/sh' as interpreter instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16311) Audit log for delete snapshot operation is missing in case of snapshot owner deleting the same
[
https://issues.apache.org/jira/browse/HBASE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456842#comment-15456842
]
Hadoop QA commented on HBASE-16311:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 52s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 97m 7s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 25s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-01 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826690/HBASE-16311-V4.patch |
| JIRA Issue | HBASE-16311 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 722b47a7181a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a1cd601 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456805#comment-15456805 ] Ted Yu commented on HBASE-15449: https://reviews.apache.org/r/51586/ > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15449: --- Attachment: 15449.v8.txt > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide
[ https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang updated HBASE-16544: - Component/s: (was: snapshots) Filesystem Integration > Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide > --- > > Key: HBASE-16544 > URL: https://issues.apache.org/jira/browse/HBASE-16544 > Project: HBase > Issue Type: Bug > Components: documentation, Filesystem Integration >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang > > reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration > (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I > think this title means that we can use S3 storage to replace HDFS, I really > tried this :(, always give me errors and hbase even can not start, see > error mentioned in jira HBASE-11045. > (2) And the details in this section are more about deploying HBase on Amazon > EC2 cluster, which has nothing to do 'using Amazon S3 storage' > (3) In all, I think we need to remove this section, or at least clarify this > section if someone fully test HBase on S3. see HBASE-15646 for more details > about this doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-15449: --- Attachment: (was: 15449.v8.txt) > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide
[ https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liang reassigned HBASE-16544: Assignee: Yi Liang > Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide > --- > > Key: HBASE-16544 > URL: https://issues.apache.org/jira/browse/HBASE-16544 > Project: HBase > Issue Type: Bug > Components: documentation, snapshots >Affects Versions: 2.0.0 >Reporter: Yi Liang >Assignee: Yi Liang > > reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration > (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I > think this title means that we can use S3 storage to replace HDFS, I really > tried this :(, always give me errors and hbase even can not start, see > error mentioned in jira HBASE-11045. > (2) And the details in this section are more about deploying HBase on Amazon > EC2 cluster, which has nothing to do 'using Amazon S3 storage' > (3) In all, I think we need to remove this section, or at least clarify this > section if someone fully test HBase on S3. see HBASE-15646 for more details > about this doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456785#comment-15456785 ] Vladimir Rodionov commented on HBASE-15449: --- [~tedyu], when you update your patch can you put it on review board? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16340) ensure no Xerces jars included
[ https://issues.apache.org/jira/browse/HBASE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456754#comment-15456754 ] Andrew Purtell commented on HBASE-16340: Pushed to 0.98, branch-1.1, and branch-1.2. I verified each change with a build using Hadoop 2.7.2. > ensure no Xerces jars included > -- > > Key: HBASE-16340 > URL: https://issues.apache.org/jira/browse/HBASE-16340 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16340.1.patch, HBASE-16340.2.patch > > > When we moved our pom to Hadoop 2.7 we picked up a transitive Xerces > implementation. We should exclude this to ensure we don't get a conflict with > the implementation that ships with the jvm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16340) ensure no Xerces jars included
[ https://issues.apache.org/jira/browse/HBASE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-16340. Resolution: Fixed > ensure no Xerces jars included > -- > > Key: HBASE-16340 > URL: https://issues.apache.org/jira/browse/HBASE-16340 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16340.1.patch, HBASE-16340.2.patch > > > When we moved our pom to Hadoop 2.7 we picked up a transitive Xerces > implementation. We should exclude this to ensure we don't get a conflict with > the implementation that ships with the jvm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456766#comment-15456766 ] Vladimir Rodionov commented on HBASE-15449: --- [~tedyu], your last patch does not apply cleanly, can you fix it? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16375) Mapreduce mini cluster using HBaseTestingUtility not setting correct resourcemanager and jobhistory webapp address of MapReduceTestingShim
[
https://issues.apache.org/jira/browse/HBASE-16375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456759#comment-15456759
]
Andrew Purtell commented on HBASE-16375:
Test failure is unrelated. Going to commit shortly.
> Mapreduce mini cluster using HBaseTestingUtility not setting correct
> resourcemanager and jobhistory webapp address of MapReduceTestingShim
>
>
> Key: HBASE-16375
> URL: https://issues.apache.org/jira/browse/HBASE-16375
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Loknath Priyatham Teja Singamsetty
>Assignee: Loknath Priyatham Teja Singamsetty
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.1.6, 0.98.22, 1.2.4
>
> Attachments: HBASE-16375.master.001.patch,
> HBASE-16375_0.98_and_above.patch,
> HBASE-16375_0.98_and_above_with_tests.patch,
> HBASE-16375_0.98_and_above_with_tests_format.patch
>
>
> Starting mapreduce mini cluster using HBaseTestingUtility is not setting
> "yarn.resourcemanager.webapp.address" and
> "mapreduce.jobhistory.webapp.address" which are required for getting the
> submitted yarn apps using mapreduce webapp. These properties are not being
> copied from jobConf of MapReduceTestingShim resulting in default values.
> {quote}
> HBaseTestingUtility.java
> // Allow the user to override FS URI for this map-reduce cluster to use.
> mrCluster = new MiniMRCluster(servers,
> FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
> null, null, new JobConf(this.conf));
> JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
> if (jobConf == null) {
> jobConf = mrCluster.createJobConf();
> }
> jobConf.set("mapreduce.cluster.local.dir",
> conf.get("mapreduce.cluster.local.dir")); //Hadoop MiniMR overwrites
> this while it should not
> LOG.info("Mini mapreduce cluster started");
> // In hadoop2, YARN/MR2 starts a mini cluster with its own conf instance
> and updates settings.
> // Our HBase MR jobs need several of these settings in order to properly
> run. So we copy the
> // necessary config properties here. YARN-129 required adding a few
> properties.
> conf.set("mapreduce.jobtracker.address",
> jobConf.get("mapreduce.jobtracker.address"));
> // this for mrv2 support; mr1 ignores this
> conf.set("mapreduce.framework.name", "yarn");
> conf.setBoolean("yarn.is.minicluster", true);
> String rmAddress = jobConf.get("yarn.resourcemanager.address");
> if (rmAddress != null) {
> conf.set("yarn.resourcemanager.address", rmAddress);
> }
> String historyAddress = jobConf.get("mapreduce.jobhistory.address");
> if (historyAddress != null) {
> conf.set("mapreduce.jobhistory.address", historyAddress);
> }
> String schedulerAddress =
> jobConf.get("yarn.resourcemanager.scheduler.address");
> if (schedulerAddress != null) {
> conf.set("yarn.resourcemanager.scheduler.address", schedulerAddress);
> }
> {quote}
> As a immediate fix for phoenix e2e test to succeed, need the below lines to
> be added as well
> {quote}
> String rmWebappAddress =
> jobConf.get("yarn.resourcemanager.webapp.address");
> if (rmWebappAddress != null) {
> conf.set("yarn.resourcemanager.webapp.address", rmWebappAddress);
> }
> String historyWebappAddress =
> jobConf.get("mapreduce.jobhistory.webapp.address");
> if (historyWebappAddress != null) {
> conf.set("mapreduce.jobhistory.webapp.address", historyWebappAddress);
> }
> {quote}
> Eventually, we should also see if we can copy over all the jobConf properties
> to HBaseTestingUtility conf object.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16340) ensure no Xerces jars included
[ https://issues.apache.org/jira/browse/HBASE-16340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16340: --- Fix Version/s: 1.2.4 1.1.7 0.98.22 > ensure no Xerces jars included > -- > > Key: HBASE-16340 > URL: https://issues.apache.org/jira/browse/HBASE-16340 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16340.1.patch, HBASE-16340.2.patch > > > When we moved our pom to Hadoop 2.7 we picked up a transitive Xerces > implementation. We should exclude this to ensure we don't get a conflict with > the implementation that ships with the jvm. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16527) IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
[ https://issues.apache.org/jira/browse/HBASE-16527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456723#comment-15456723 ] Hudson commented on HBASE-16527: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1523 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1523/]) HBASE-16527 IOExceptions from DFS client still can cause CatalogJanitor (tedyu: rev a1cd601ad7199ea91e9f9b5e462c24d6681bc870) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > IOExceptions from DFS client still can cause CatalogJanitor to delete > referenced files > -- > > Key: HBASE-16527 > URL: https://issues.apache.org/jira/browse/HBASE-16527 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.22, 1.1.7, 1.2.4 > > Attachments: HBASE-16527-v1.patch, HBASE-16527-v2.patch > > > that was fixed partially in HBASE-13331, but issue still exists , now a > little bit deeper in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16497) Add test for backup / restore involving MOB table
[ https://issues.apache.org/jira/browse/HBASE-16497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16497: --- Labels: backup mob (was: backup) > Add test for backup / restore involving MOB table > - > > Key: HBASE-16497 > URL: https://issues.apache.org/jira/browse/HBASE-16497 > Project: HBase > Issue Type: Test >Reporter: Ted Yu > Labels: backup, mob > > Currently backup / restore tests only deal with non-MOB tables. > This issue is to add coverage for backup / restore of table with MOB column > family. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16506) Use subprocedure of Proc V2 for snapshot in BackupProcedure
[ https://issues.apache.org/jira/browse/HBASE-16506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16506: --- Labels: backup snapshot (was: backup) > Use subprocedure of Proc V2 for snapshot in BackupProcedure > --- > > Key: HBASE-16506 > URL: https://issues.apache.org/jira/browse/HBASE-16506 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu > Labels: backup, snapshot > > Currently for SNAPSHOT_TABLES stage, we loop through the tables and take > snapshot for each table. > If the master restarts in the middle of this stage, we would restart taking > snapshot from the first table. > This issue would use subprocedure for each snapshot so that we don't need to > take snapshot for the table(s) whose snapshot is complete before the master > restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16538) Version mismatch in HBaseConfiguration.checkDefaultsVersion
[
https://issues.apache.org/jira/browse/HBASE-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456612#comment-15456612
]
Andrew Purtell commented on HBASE-16538:
bq. I think the root cause is order of processing of static block and
annotations.
Sigh. I suppose this can vary between JVM vendor and version.
> Version mismatch in HBaseConfiguration.checkDefaultsVersion
> ---
>
> Key: HBASE-16538
> URL: https://issues.apache.org/jira/browse/HBASE-16538
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
> Labels: configuration, test-failure
>
> {noformat}
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures
> testYieldEachExecutionStep(org.apache.hadoop.hbase.procedure2.TestYieldProcedures)
> Time elapsed: 0.255 sec <<< ERROR!
> java.lang.RuntimeException: hbase-default.xml file seems to be for an older
> version of HBase (2.0.0-SNAPSHOT), this version is Unknown
> at
> org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:73)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:83)
> at
> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
> at
> org.apache.hadoop.hbase.HBaseCommonTestingUtility.(HBaseCommonTestingUtility.java:46)
> at
> org.apache.hadoop.hbase.procedure2.TestYieldProcedures.setUp(TestYieldProcedures.java:63)
> {noformat}
> (Exact test is not important)
> Reference run:
> https://builds.apache.org/view/All/job/HBase-Trunk_matrix/jdk=JDK%201.8%20(latest),label=yahoo-not-h2/1515/console
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456553#comment-15456553 ] huaxiang sun commented on HBASE-16345: -- Seems that rpc server had no response within 60 seconds, and this caused rpc timeout. Seems flaky to me. Reattach the patch to trigger a new run as it passed locally for 5 runs. > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions
[ https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456498#comment-15456498 ] huaxiang sun commented on HBASE-16345: -- Local run of TestRestoreSnapshotFromClient#testRestoreSchemaChange passed. I am looking at the stack trace to see what is going on. > RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer > Exceptions > -- > > Key: HBASE-16345 > URL: https://issues.apache.org/jira/browse/HBASE-16345 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-16345-v001.patch, HBASE-16345.master.001.patch, > HBASE-16345.master.002.patch, HBASE-16345.master.003.patch > > > Update for the description. Debugged more at this front based on the comments > from Enis. > The cause is that for the primary replica, if its retry is exhausted too > fast, f.get() [1] returns ExecutionException. This Exception needs to be > ignored and continue with the replicas. > The other issue is that after adding calls for the replicas, if the first > completed task gets ExecutionException (due to the retry exhausted), it > throws the exception to the client[2]. > In this case, it needs to loop through these tasks, waiting for the success > one. If no one succeeds, throw exception. > Similar for the scan as well > [1] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197 > [2] > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16465) Disable region splits and merges, balancer during full backup
[
https://issues.apache.org/jira/browse/HBASE-16465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456497#comment-15456497
]
Ted Yu commented on HBASE-16465:
{code}
+ private boolean safeMode = true;
{code}
Mode implies enum. Please rename the variable.
{code}
+boolean balancer = false;
{code}
Please name the variable balancerEnabled.
{code}
+if (isFullBackup && userRequest.isSafeMode()) {
{code}
Should default value for safe mode be off ?
For testFullBackupMultipleCommandUnsafe, I don't see induced split / merge.
> Disable region splits and merges, balancer during full backup
> -
>
> Key: HBASE-16465
> URL: https://issues.apache.org/jira/browse/HBASE-16465
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Attachments: HBASE-16465-v1.patch, HBASE-16465-v2.patch,
> HBASE-16465-v3.patch, HBASE-16465-v4.patch
>
>
> Incorporate HBASE-15128
> Balancer, catalog janitor and region normalizer should be disabled as well
> during full backup
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16311) Audit log for delete snapshot operation is missing in case of snapshot owner deleting the same
[
https://issues.apache.org/jira/browse/HBASE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liang updated HBASE-16311:
-
Attachment: HBASE-16311-V4.patch
> Audit log for delete snapshot operation is missing in case of snapshot owner
> deleting the same
> --
>
> Key: HBASE-16311
> URL: https://issues.apache.org/jira/browse/HBASE-16311
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Abhishek Kumar
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16311-V1.patch, HBASE-16311-V2.patch,
> HBASE-16311-V3.patch, HBASE-16311-V4.patch
>
>
> 1. Audit log seems to be left as a TODO task in AccessController.java:
> {code}
> @Override
> public void preDeleteSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot) throws IOException {
> if (SnapshotDescriptionUtils.isSnapshotOwner(snapshot, getActiveUser())) {
> // Snapshot owner is allowed to delete the snapshot
> // TODO: We are not logging this for audit
> } else {
> requirePermission("deleteSnapshot", Action.ADMIN);
> }
> }
> {code}
> 2. Also, snapshot name is not getting logged in the audit logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16311) Audit log for delete snapshot operation is missing in case of snapshot owner deleting the same
[
https://issues.apache.org/jira/browse/HBASE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456492#comment-15456492
]
Yi Liang commented on HBASE-16311:
--
thank for reviewing [~jerryhe], yes, the snapshot owner may not have global
Action.ADMIN permission, so we can just set it as null
> Audit log for delete snapshot operation is missing in case of snapshot owner
> deleting the same
> --
>
> Key: HBASE-16311
> URL: https://issues.apache.org/jira/browse/HBASE-16311
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
>Affects Versions: 2.0.0
>Reporter: Abhishek Kumar
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16311-V1.patch, HBASE-16311-V2.patch,
> HBASE-16311-V3.patch
>
>
> 1. Audit log seems to be left as a TODO task in AccessController.java:
> {code}
> @Override
> public void preDeleteSnapshot(final
> ObserverContext ctx,
> final SnapshotDescription snapshot) throws IOException {
> if (SnapshotDescriptionUtils.isSnapshotOwner(snapshot, getActiveUser())) {
> // Snapshot owner is allowed to delete the snapshot
> // TODO: We are not logging this for audit
> } else {
> requirePermission("deleteSnapshot", Action.ADMIN);
> }
> }
> {code}
> 2. Also, snapshot name is not getting logged in the audit logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16546) please ignore
[ https://issues.apache.org/jira/browse/HBASE-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dima Spivak resolved HBASE-16546. - Resolution: Invalid No JIRA tests, please. :) > please ignore > - > > Key: HBASE-16546 > URL: https://issues.apache.org/jira/browse/HBASE-16546 > Project: HBase > Issue Type: Bug >Reporter: Joe Programmer > > test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16546) please ignore
Joe Programmer created HBASE-16546: -- Summary: please ignore Key: HBASE-16546 URL: https://issues.apache.org/jira/browse/HBASE-16546 Project: HBase Issue Type: Bug Reporter: Joe Programmer test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16366) Restore operation into new table may fail
[
https://issues.apache.org/jira/browse/HBASE-16366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16366:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Fixed spacing around:
{code}
+ if (EnvironmentEdgeManager.currentTime() - startTime >
TABLE_AVAILABILITY_WAIT_TIME) {
{code}
Thanks for the patch, Vlad.
> Restore operation into new table may fail
> -
>
> Key: HBASE-16366
> URL: https://issues.apache.org/jira/browse/HBASE-16366
> Project: HBase
> Issue Type: Bug
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16366-v1.patch, HBASE-16366-v2.patch
>
>
> If restore from backup into new table we need to make sure that new table is
> available online.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
