[hadoop] branch branch-2.10 updated: HADOOP-17885. Upgrade JSON smart to 1.3.3 on branch-2.10 (#3370)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 42599f6 HADOOP-17885. Upgrade JSON smart to 1.3.3 on branch-2.10
(#3370)
42599f6 is described below
commit 42599f6c8f342d015b5225d6756b3f72a231da4b
Author: Ahmed Hussein <50450311+amahuss...@users.noreply.github.com>
AuthorDate: Thu Sep 2 16:40:06 2021 -0500
HADOOP-17885. Upgrade JSON smart to 1.3.3 on branch-2.10 (#3370)
---
hadoop-project/pom.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index da848de..0245444 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -147,6 +147,7 @@
${project.version}
+1.3.3
5.3.0
1.10.11
@@ -1218,7 +1219,7 @@
net.minidev
json-smart
-1.3.1
+${json-smart.version}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.10 updated: HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new d636420 HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
d636420 is described below
commit d636420e7c4222e7ed1339fe1c19a4ebef1e5f9f
Author: Ahmed Hussein <50450311+amahuss...@users.noreply.github.com>
AuthorDate: Thu Sep 2 16:11:07 2021 -0500
HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
---
hadoop-project/pom.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 7f356e0..da848de 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -148,6 +148,7 @@
so that Hadoop 3.x can declare itself a 2.x artifact. -->
${project.version}
5.3.0
+1.10.11
@@ -971,7 +972,7 @@
org.apache.ant
ant
-1.10.9
+${apache-ant.version}
com.google.re2j
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 601591c HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
601591c is described below
commit 601591c02144c9c0dfed2c4011187dc7bb329157
Author: Ahmed Hussein <50450311+amahuss...@users.noreply.github.com>
AuthorDate: Thu Sep 2 16:11:07 2021 -0500
HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
---
hadoop-project/pom.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 409c380..810add8 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -188,6 +188,7 @@
5.3.0
2.4.7
9.8.1
+1.10.11
@@ -1122,7 +1123,7 @@
org.apache.ant
ant
-1.10.9
+${apache-ant.version}
com.google.re2j
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new fc5d67d HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
fc5d67d is described below
commit fc5d67dfb48b9748e1c7c646657c824939506fe9
Author: Ahmed Hussein <50450311+amahuss...@users.noreply.github.com>
AuthorDate: Thu Sep 2 16:11:07 2021 -0500
HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
---
hadoop-project/pom.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 715b1dd..53937c6 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -212,6 +212,7 @@
5.3.0
2.4.7
9.8.1
+1.10.11
@@ -1205,7 +1206,7 @@
org.apache.ant
ant
-1.10.9
+${apache-ant.version}
com.google.re2j
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 0512073 HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
0512073 is described below
commit 051207375b30ffa126cf9c961ee97535c1dfabb1
Author: Ahmed Hussein <50450311+amahuss...@users.noreply.github.com>
AuthorDate: Thu Sep 2 16:11:07 2021 -0500
HADOOP-17886. Upgrade ant to 1.10.11 (#3371)
---
hadoop-project/pom.xml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 3d6dfb8..1435eae2 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -211,6 +211,7 @@
5.3.0
2.4.7
9.8.1
+1.10.11
@@ -1238,7 +1239,7 @@
org.apache.ant
ant
-1.10.9
+${apache-ant.version}
com.google.re2j
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.10 updated: YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new b79cd11 YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails
with NPE. Contributed by Prabhu Joseph.
b79cd11 is described below
commit b79cd11fcd4d43c731a1a23cf79d38b0cecacaa5
Author: Abhishek Modi
AuthorDate: Tue Aug 13 19:04:00 2019 +0530
YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE.
Contributed by Prabhu Joseph.
(cherry picked from commit b4097b96a39bad6214b01989e7f2fb37dad70793)
---
.../timeline/RollingLevelDBTimelineStore.java | 55 --
1 file changed, 29 insertions(+), 26 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
index 1ac170c..5087db4 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
@@ -791,39 +791,42 @@ public class RollingLevelDBTimelineStore extends
AbstractService implements
entity = getEntity(entityId, entityType, startTime, queryFields,
iterator, key, kp.getOffset());
}
- // determine if the retrieved entity matches the provided secondary
- // filters, and if so add it to the list of entities to return
- boolean filterPassed = true;
- if (secondaryFilters != null) {
-for (NameValuePair filter : secondaryFilters) {
- Object v = entity.getOtherInfo().get(filter.getName());
- if (v == null) {
-Set vs = entity.getPrimaryFilters()
-.get(filter.getName());
-if (vs == null || !vs.contains(filter.getValue())) {
+
+ if (entity != null) {
+// determine if the retrieved entity matches the provided secondary
+// filters, and if so add it to the list of entities to return
+boolean filterPassed = true;
+if (secondaryFilters != null) {
+ for (NameValuePair filter : secondaryFilters) {
+Object v = entity.getOtherInfo().get(filter.getName());
+if (v == null) {
+ Set vs = entity.getPrimaryFilters()
+ .get(filter.getName());
+ if (vs == null || !vs.contains(filter.getValue())) {
+filterPassed = false;
+break;
+ }
+} else if (!v.equals(filter.getValue())) {
filterPassed = false;
break;
}
- } else if (!v.equals(filter.getValue())) {
-filterPassed = false;
-break;
}
}
- }
- if (filterPassed) {
-if (entity.getDomainId() == null) {
- entity.setDomainId(DEFAULT_DOMAIN_ID);
-}
-if (checkAcl == null || checkAcl.check(entity)) {
- // Remove primary filter and other info if they are added for
- // matching secondary filters
- if (addPrimaryFilters) {
-entity.setPrimaryFilters(null);
+if (filterPassed) {
+ if (entity.getDomainId() == null) {
+entity.setDomainId(DEFAULT_DOMAIN_ID);
}
- if (addOtherInfo) {
-entity.setOtherInfo(null);
+ if (checkAcl == null || checkAcl.check(entity)) {
+// Remove primary filter and other info if they are added for
+// matching secondary filters
+if (addPrimaryFilters) {
+ entity.setPrimaryFilters(null);
+}
+if (addOtherInfo) {
+ entity.setOtherInfo(null);
+}
+entities.addEntity(entity);
}
- entities.addEntity(entity);
}
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph.
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 6a02596 YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails
with NPE. Contributed by Prabhu Joseph.
6a02596 is described below
commit 6a02596f428b564f388d0cf6cc5b13ab54520f2d
Author: Abhishek Modi
AuthorDate: Tue Aug 13 19:04:00 2019 +0530
YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE.
Contributed by Prabhu Joseph.
(cherry picked from commit b4097b96a39bad6214b01989e7f2fb37dad70793)
---
.../timeline/RollingLevelDBTimelineStore.java | 55 --
1 file changed, 29 insertions(+), 26 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
index 255547b..cad181a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/RollingLevelDBTimelineStore.java
@@ -797,39 +797,42 @@ public class RollingLevelDBTimelineStore extends
AbstractService implements
entity = getEntity(entityId, entityType, startTime, queryFields,
iterator, key, kp.getOffset());
}
- // determine if the retrieved entity matches the provided secondary
- // filters, and if so add it to the list of entities to return
- boolean filterPassed = true;
- if (secondaryFilters != null) {
-for (NameValuePair filter : secondaryFilters) {
- Object v = entity.getOtherInfo().get(filter.getName());
- if (v == null) {
-Set vs = entity.getPrimaryFilters()
-.get(filter.getName());
-if (vs == null || !vs.contains(filter.getValue())) {
+
+ if (entity != null) {
+// determine if the retrieved entity matches the provided secondary
+// filters, and if so add it to the list of entities to return
+boolean filterPassed = true;
+if (secondaryFilters != null) {
+ for (NameValuePair filter : secondaryFilters) {
+Object v = entity.getOtherInfo().get(filter.getName());
+if (v == null) {
+ Set vs = entity.getPrimaryFilters()
+ .get(filter.getName());
+ if (vs == null || !vs.contains(filter.getValue())) {
+filterPassed = false;
+break;
+ }
+} else if (!v.equals(filter.getValue())) {
filterPassed = false;
break;
}
- } else if (!v.equals(filter.getValue())) {
-filterPassed = false;
-break;
}
}
- }
- if (filterPassed) {
-if (entity.getDomainId() == null) {
- entity.setDomainId(DEFAULT_DOMAIN_ID);
-}
-if (checkAcl == null || checkAcl.check(entity)) {
- // Remove primary filter and other info if they are added for
- // matching secondary filters
- if (addPrimaryFilters) {
-entity.setPrimaryFilters(null);
+if (filterPassed) {
+ if (entity.getDomainId() == null) {
+entity.setDomainId(DEFAULT_DOMAIN_ID);
}
- if (addOtherInfo) {
-entity.setOtherInfo(null);
+ if (checkAcl == null || checkAcl.check(entity)) {
+// Remove primary filter and other info if they are added for
+// matching secondary filters
+if (addPrimaryFilters) {
+ entity.setPrimaryFilters(null);
+}
+if (addOtherInfo) {
+ entity.setOtherInfo(null);
+}
+entities.addEntity(entity);
}
- entities.addEntity(entity);
}
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new b7b9bd3 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
b7b9bd3 is described below
commit b7b9bd32db269de6759dde4126cf5d2c28fdabd7
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 11:39:06 2020 -0500
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
---
.../src/main/resources/checkstyle/checkstyle.xml | 2 +-
.../hadoop/metrics2/impl/MetricsRecords.java | 26 +---
.../metrics2/impl/TestMetricsSystemImpl.java | 30 +++--
.../blockmanagement/CombinedHostFileManager.java | 78 --
.../server/namenode/NameNodeResourceChecker.java | 24 +++
.../hdfs/server/namenode/snapshot/Snapshot.java| 19 ++
.../yarn/logaggregation/AggregatedLogFormat.java | 12 +---
.../LogAggregationFileController.java | 22 +++---
.../ifile/LogAggregationIndexedFileController.java | 21 +++---
.../logaggregation/AppLogAggregatorImpl.java | 16 ++---
10 files changed, 91 insertions(+), 159 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index e0a55f7..3e6ae7a 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -123,7 +123,7 @@
+value="^com\.google\.common\.base\.(Optional|Function|Predicate),
^com\.google\.common\.collect\.(ImmutableListMultimap)"/>
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
index 5d52cad..7865714 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
@@ -18,8 +18,8 @@
package org.apache.hadoop.metrics2.impl;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Iterables;
+import java.util.function.Predicate;
+import java.util.stream.StreamSupport;
import org.apache.hadoop.metrics2.AbstractMetric;
import org.apache.hadoop.metrics2.MetricsRecord;
import org.apache.hadoop.metrics2.MetricsTag;
@@ -65,16 +65,22 @@ public class MetricsRecords {
resourceLimitMetric);
}
- private static MetricsTag getFirstTagByName(MetricsRecord record, String
name) {
-return Iterables.getFirst(Iterables.filter(record.tags(),
-new MetricsTagPredicate(name)), null);
+ private static MetricsTag getFirstTagByName(MetricsRecord record,
+ String name) {
+if (record.tags() == null) {
+ return null;
+}
+return record.tags().stream().filter(
+new MetricsTagPredicate(name)).findFirst().orElse(null);
}
private static AbstractMetric getFirstMetricByName(
MetricsRecord record, String name) {
-return Iterables.getFirst(
-Iterables.filter(record.metrics(), new AbstractMetricPredicate(name)),
-null);
+if (record.metrics() == null) {
+ return null;
+}
+return StreamSupport.stream(record.metrics().spliterator(), false)
+.filter(new AbstractMetricPredicate(name)).findFirst().orElse(null);
}
private static class MetricsTagPredicate implements Predicate {
@@ -86,7 +92,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(MetricsTag input) {
+public boolean test(MetricsTag input) {
return input.name().equals(tagName);
}
}
@@ -101,7 +107,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(AbstractMetric input) {
+public boolean test(AbstractMetric input) {
return input.name().equals(metricName);
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
index f3a2553..41b29a0 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
@@ -23,9 +23,7 @@ import java.io.IOException;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
-
-import javax.annotation.Nullable;
-
+import java.util.stream.StreamSup
[hadoop] branch branch-3.2 updated: HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 8fd3dcc HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
8fd3dcc is described below
commit 8fd3dcc9ce75142b80d902693e7df518963ee690
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 11:39:06 2020 -0500
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
---
.../src/main/resources/checkstyle/checkstyle.xml | 2 +-
.../hadoop/metrics2/impl/MetricsRecords.java | 26 +---
.../metrics2/impl/TestMetricsSystemImpl.java | 30 +++--
.../blockmanagement/CombinedHostFileManager.java | 78 --
.../server/namenode/NameNodeResourceChecker.java | 25 +++
.../hdfs/server/namenode/snapshot/Snapshot.java| 19 ++
.../yarn/logaggregation/AggregatedLogFormat.java | 12 +---
.../LogAggregationFileController.java | 22 +++---
.../ifile/LogAggregationIndexedFileController.java | 21 +++---
.../logaggregation/AppLogAggregatorImpl.java | 16 ++---
10 files changed, 91 insertions(+), 160 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index e0a55f7..3e6ae7a 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -123,7 +123,7 @@
+value="^com\.google\.common\.base\.(Optional|Function|Predicate),
^com\.google\.common\.collect\.(ImmutableListMultimap)"/>
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
index 5d52cad..7865714 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
@@ -18,8 +18,8 @@
package org.apache.hadoop.metrics2.impl;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Iterables;
+import java.util.function.Predicate;
+import java.util.stream.StreamSupport;
import org.apache.hadoop.metrics2.AbstractMetric;
import org.apache.hadoop.metrics2.MetricsRecord;
import org.apache.hadoop.metrics2.MetricsTag;
@@ -65,16 +65,22 @@ public class MetricsRecords {
resourceLimitMetric);
}
- private static MetricsTag getFirstTagByName(MetricsRecord record, String
name) {
-return Iterables.getFirst(Iterables.filter(record.tags(),
-new MetricsTagPredicate(name)), null);
+ private static MetricsTag getFirstTagByName(MetricsRecord record,
+ String name) {
+if (record.tags() == null) {
+ return null;
+}
+return record.tags().stream().filter(
+new MetricsTagPredicate(name)).findFirst().orElse(null);
}
private static AbstractMetric getFirstMetricByName(
MetricsRecord record, String name) {
-return Iterables.getFirst(
-Iterables.filter(record.metrics(), new AbstractMetricPredicate(name)),
-null);
+if (record.metrics() == null) {
+ return null;
+}
+return StreamSupport.stream(record.metrics().spliterator(), false)
+.filter(new AbstractMetricPredicate(name)).findFirst().orElse(null);
}
private static class MetricsTagPredicate implements Predicate {
@@ -86,7 +92,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(MetricsTag input) {
+public boolean test(MetricsTag input) {
return input.name().equals(tagName);
}
}
@@ -101,7 +107,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(AbstractMetric input) {
+public boolean test(AbstractMetric input) {
return input.name().equals(metricName);
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
index 47520b5..c648a43 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
@@ -23,9 +23,7 @@ import java.io.IOException;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
-
-import javax.annotation.Nullable;
-
+import java.util.stream.StreamSup
[hadoop] branch branch-3.3 updated: HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 9e7266d HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
9e7266d is described below
commit 9e7266df6c5b77e7ab8cd13d2f588eb109cd2bda
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 11:39:06 2020 -0500
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
---
.../src/main/resources/checkstyle/checkstyle.xml | 2 +-
.../hadoop/metrics2/impl/MetricsRecords.java | 26 +---
.../metrics2/impl/TestMetricsSystemImpl.java | 30 +++--
.../blockmanagement/CombinedHostFileManager.java | 78 --
.../server/namenode/NameNodeResourceChecker.java | 25 +++
.../hdfs/server/namenode/snapshot/Snapshot.java| 19 ++
.../yarn/logaggregation/AggregatedLogFormat.java | 12 +---
.../LogAggregationFileController.java | 23 +++
.../ifile/LogAggregationIndexedFileController.java | 21 +++---
.../logaggregation/AppLogAggregatorImpl.java | 16 ++---
10 files changed, 91 insertions(+), 161 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index e0a55f7..3e6ae7a 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -123,7 +123,7 @@
+value="^com\.google\.common\.base\.(Optional|Function|Predicate),
^com\.google\.common\.collect\.(ImmutableListMultimap)"/>
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
index 5d52cad..7865714 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
@@ -18,8 +18,8 @@
package org.apache.hadoop.metrics2.impl;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Iterables;
+import java.util.function.Predicate;
+import java.util.stream.StreamSupport;
import org.apache.hadoop.metrics2.AbstractMetric;
import org.apache.hadoop.metrics2.MetricsRecord;
import org.apache.hadoop.metrics2.MetricsTag;
@@ -65,16 +65,22 @@ public class MetricsRecords {
resourceLimitMetric);
}
- private static MetricsTag getFirstTagByName(MetricsRecord record, String
name) {
-return Iterables.getFirst(Iterables.filter(record.tags(),
-new MetricsTagPredicate(name)), null);
+ private static MetricsTag getFirstTagByName(MetricsRecord record,
+ String name) {
+if (record.tags() == null) {
+ return null;
+}
+return record.tags().stream().filter(
+new MetricsTagPredicate(name)).findFirst().orElse(null);
}
private static AbstractMetric getFirstMetricByName(
MetricsRecord record, String name) {
-return Iterables.getFirst(
-Iterables.filter(record.metrics(), new AbstractMetricPredicate(name)),
-null);
+if (record.metrics() == null) {
+ return null;
+}
+return StreamSupport.stream(record.metrics().spliterator(), false)
+.filter(new AbstractMetricPredicate(name)).findFirst().orElse(null);
}
private static class MetricsTagPredicate implements Predicate {
@@ -86,7 +92,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(MetricsTag input) {
+public boolean test(MetricsTag input) {
return input.name().equals(tagName);
}
}
@@ -101,7 +107,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(AbstractMetric input) {
+public boolean test(AbstractMetric input) {
return input.name().equals(metricName);
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
index 1b40a17..fd6f167 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
@@ -23,9 +23,7 @@ import java.io.IOException;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
-
-import javax.annotation.Nullable;
-
+import java.util.stream.StreamSup
[hadoop] branch trunk updated: HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 1f71c4a HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
1f71c4a is described below
commit 1f71c4ae71427a8a7476eaef64187a5643596552
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 11:39:06 2020 -0500
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles
---
.../src/main/resources/checkstyle/checkstyle.xml | 2 +-
.../hadoop/metrics2/impl/MetricsRecords.java | 26 +---
.../metrics2/impl/TestMetricsSystemImpl.java | 30 +++--
.../blockmanagement/CombinedHostFileManager.java | 78 --
.../server/namenode/NameNodeResourceChecker.java | 25 +++
.../hdfs/server/namenode/snapshot/Snapshot.java| 19 ++
.../yarn/logaggregation/AggregatedLogFormat.java | 12 +---
.../LogAggregationFileController.java | 23 +++
.../ifile/LogAggregationIndexedFileController.java | 21 +++---
.../logaggregation/AppLogAggregatorImpl.java | 16 ++---
10 files changed, 91 insertions(+), 161 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index e0a55f7..3e6ae7a 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -123,7 +123,7 @@
+value="^com\.google\.common\.base\.(Optional|Function|Predicate),
^com\.google\.common\.collect\.(ImmutableListMultimap)"/>
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
index 5d52cad..7865714 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/MetricsRecords.java
@@ -18,8 +18,8 @@
package org.apache.hadoop.metrics2.impl;
-import com.google.common.base.Predicate;
-import com.google.common.collect.Iterables;
+import java.util.function.Predicate;
+import java.util.stream.StreamSupport;
import org.apache.hadoop.metrics2.AbstractMetric;
import org.apache.hadoop.metrics2.MetricsRecord;
import org.apache.hadoop.metrics2.MetricsTag;
@@ -65,16 +65,22 @@ public class MetricsRecords {
resourceLimitMetric);
}
- private static MetricsTag getFirstTagByName(MetricsRecord record, String
name) {
-return Iterables.getFirst(Iterables.filter(record.tags(),
-new MetricsTagPredicate(name)), null);
+ private static MetricsTag getFirstTagByName(MetricsRecord record,
+ String name) {
+if (record.tags() == null) {
+ return null;
+}
+return record.tags().stream().filter(
+new MetricsTagPredicate(name)).findFirst().orElse(null);
}
private static AbstractMetric getFirstMetricByName(
MetricsRecord record, String name) {
-return Iterables.getFirst(
-Iterables.filter(record.metrics(), new AbstractMetricPredicate(name)),
-null);
+if (record.metrics() == null) {
+ return null;
+}
+return StreamSupport.stream(record.metrics().spliterator(), false)
+.filter(new AbstractMetricPredicate(name)).findFirst().orElse(null);
}
private static class MetricsTagPredicate implements Predicate {
@@ -86,7 +92,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(MetricsTag input) {
+public boolean test(MetricsTag input) {
return input.name().equals(tagName);
}
}
@@ -101,7 +107,7 @@ public class MetricsRecords {
}
@Override
-public boolean apply(AbstractMetric input) {
+public boolean test(AbstractMetric input) {
return input.name().equals(metricName);
}
}
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
index 1b40a17..fd6f167 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/impl/TestMetricsSystemImpl.java
@@ -23,9 +23,7 @@ import java.io.IOException;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
-
-import javax.annotation.Nullable;
-
+import java.util.stream.StreamSupport;
import org.junit.Test;
import org.junit.runner.RunWith;
@@ -38,7 +3
[hadoop] branch branch-3.1 updated: HADOOP-17101. Replace Guava Function with Java8+ Function
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 34a6cbb HADOOP-17101. Replace Guava Function with Java8+ Function
34a6cbb is described below
commit 34a6cbb5bbedf53d618f3b3e3522f80e418e476c
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 09:53:18 2020 -0500
HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
(cherry picked from commit 43a865dc07437e0f767a902c4cfe6ca0472bf57f)
---
.../src/main/resources/checkstyle/checkstyle.xml | 7 +-
.../hdfs/server/blockmanagement/HostSet.java | 25 ++--
.../hadoop/hdfs/server/namenode/JournalSet.java| 22 +-
.../hadoop/hdfs/server/protocol/RemoteEditLog.java | 11 -
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 17 +-
.../mapreduce/lib/input/TestFileInputFormat.java | 27 --
.../impl/pb/GetApplicationsRequestPBImpl.java | 11 ++---
7 files changed, 50 insertions(+), 70 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index 8f3d3f1..e0a55f7 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -119,7 +119,12 @@
-
+
+
+
+
+
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
index 958557b..16f67ee 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
@@ -17,15 +17,12 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
+
import com.google.common.base.Preconditions;
import com.google.common.collect.HashMultimap;
-import com.google.common.collect.Iterators;
import com.google.common.collect.Multimap;
import com.google.common.collect.UnmodifiableIterator;
-import javax.annotation.Nullable;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.Collection;
@@ -101,14 +98,16 @@ public class HostSet implements
Iterable {
@Override
public String toString() {
StringBuilder sb = new StringBuilder("HostSet(");
-Joiner.on(",").appendTo(sb, Iterators.transform(iterator(),
-new Function() {
- @Override
- public String apply(@Nullable InetSocketAddress addr) {
-assert addr != null;
-return addr.getAddress().getHostAddress() + ":" + addr.getPort();
- }
-}));
-return sb.append(")").toString();
+Iterator iter = iterator();
+String sep = "";
+while (iter.hasNext()) {
+ InetSocketAddress addr = iter.next();
+ sb.append(sep);
+ sb.append(addr.getAddress().getHostAddress());
+ sb.append(':');
+ sb.append(addr.getPort());
+ sep = ",";
+}
+return sb.append(')').toString();
}
}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
index e7f2adb..4b50531 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
@@ -24,8 +24,10 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
+import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.PriorityQueue;
import java.util.SortedSet;
import java.util.concurrent.CopyOnWriteArrayList;
@@ -38,12 +40,8 @@ import org.apache.hadoop.hdfs.server.common.StorageInfo;
import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLog;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest;
-
import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableListMultimap;
import com.google.common.collect.Lists;
-import com.google.common.collect.Multimaps;
im
[hadoop] branch branch-3.2 updated: HADOOP-17101. Replace Guava Function with Java8+ Function
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 43a865d HADOOP-17101. Replace Guava Function with Java8+ Function
43a865d is described below
commit 43a865dc07437e0f767a902c4cfe6ca0472bf57f
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 09:53:18 2020 -0500
HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
---
.../src/main/resources/checkstyle/checkstyle.xml | 7 +-
.../hdfs/server/blockmanagement/HostSet.java | 25 ++--
.../hadoop/hdfs/server/namenode/JournalSet.java| 22 +-
.../hadoop/hdfs/server/protocol/RemoteEditLog.java | 11 -
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 16 -
.../mapreduce/lib/input/TestFileInputFormat.java | 27 --
.../impl/pb/GetApplicationsRequestPBImpl.java | 12 ++
7 files changed, 49 insertions(+), 71 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index 8f3d3f1..e0a55f7 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -119,7 +119,12 @@
-
+
+
+
+
+
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
index 958557b..16f67ee 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
@@ -17,15 +17,12 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
+
import com.google.common.base.Preconditions;
import com.google.common.collect.HashMultimap;
-import com.google.common.collect.Iterators;
import com.google.common.collect.Multimap;
import com.google.common.collect.UnmodifiableIterator;
-import javax.annotation.Nullable;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.Collection;
@@ -101,14 +98,16 @@ public class HostSet implements
Iterable {
@Override
public String toString() {
StringBuilder sb = new StringBuilder("HostSet(");
-Joiner.on(",").appendTo(sb, Iterators.transform(iterator(),
-new Function() {
- @Override
- public String apply(@Nullable InetSocketAddress addr) {
-assert addr != null;
-return addr.getAddress().getHostAddress() + ":" + addr.getPort();
- }
-}));
-return sb.append(")").toString();
+Iterator iter = iterator();
+String sep = "";
+while (iter.hasNext()) {
+ InetSocketAddress addr = iter.next();
+ sb.append(sep);
+ sb.append(addr.getAddress().getHostAddress());
+ sb.append(':');
+ sb.append(addr.getPort());
+ sep = ",";
+}
+return sb.append(')').toString();
}
}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
index 7be7073..51f8b4f 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
@@ -24,8 +24,10 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
+import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.PriorityQueue;
import java.util.SortedSet;
import java.util.concurrent.CopyOnWriteArrayList;
@@ -38,12 +40,8 @@ import org.apache.hadoop.hdfs.server.common.StorageInfo;
import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLog;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest;
-
import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableListMultimap;
import com.google.common.collect.Lists;
-import com.google.common.collect.Multimaps;
import com.google.common.collect.Sets;
/**
@@ -628,7 +626,7 @@ public c
[hadoop] branch branch-3.3 updated: HADOOP-17101. Replace Guava Function with Java8+ Function
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 5969922 HADOOP-17101. Replace Guava Function with Java8+ Function
5969922 is described below
commit 5969922305204a04e567609550ef99209e36bb5c
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 09:53:18 2020 -0500
HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
---
.../src/main/resources/checkstyle/checkstyle.xml | 2 +-
.../hdfs/server/blockmanagement/HostSet.java | 25 ++--
.../hadoop/hdfs/server/namenode/JournalSet.java| 22 +-
.../hadoop/hdfs/server/protocol/RemoteEditLog.java | 11 -
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 16 -
.../mapreduce/lib/input/TestFileInputFormat.java | 27 --
.../impl/pb/GetApplicationsRequestPBImpl.java | 12 ++
7 files changed, 44 insertions(+), 71 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index 54a5943..e0a55f7 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -123,7 +123,7 @@
+value="^com\.google\.common\.base\.(Optional|Function),
^com\.google\.common\.collect\.(ImmutableListMultimap)"/>
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
index cf7cfac..1d16cae 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
@@ -17,15 +17,12 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
+
import com.google.common.base.Preconditions;
import com.google.common.collect.HashMultimap;
-import com.google.common.collect.Iterators;
import com.google.common.collect.Multimap;
import com.google.common.collect.UnmodifiableIterator;
-import javax.annotation.Nullable;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.Collection;
@@ -101,14 +98,16 @@ public class HostSet implements
Iterable {
@Override
public String toString() {
StringBuilder sb = new StringBuilder("HostSet(");
-Joiner.on(",").appendTo(sb, Iterators.transform(iterator(),
-new Function() {
- @Override
- public String apply(@Nullable InetSocketAddress addr) {
-assert addr != null;
-return addr.getAddress().getHostAddress() + ":" + addr.getPort();
- }
-}));
-return sb.append(")").toString();
+Iterator iter = iterator();
+String sep = "";
+while (iter.hasNext()) {
+ InetSocketAddress addr = iter.next();
+ sb.append(sep);
+ sb.append(addr.getAddress().getHostAddress());
+ sb.append(':');
+ sb.append(addr.getPort());
+ sep = ",";
+}
+return sb.append(')').toString();
}
}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
index a5df9f5..eb3a140 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
@@ -24,8 +24,10 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
+import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.PriorityQueue;
import java.util.SortedSet;
import java.util.concurrent.CopyOnWriteArrayList;
@@ -38,13 +40,9 @@ import org.apache.hadoop.hdfs.server.common.StorageInfo;
import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLog;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest;
-
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableListMultimap;
import com
[hadoop] branch trunk updated: HADOOP-17101. Replace Guava Function with Java8+ Function
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 98fcffe HADOOP-17101. Replace Guava Function with Java8+ Function
98fcffe is described below
commit 98fcffe93f9ef910654574f69591fcdc621523af
Author: Ahmed Hussein
AuthorDate: Wed Jul 15 09:53:18 2020 -0500
HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles
---
.../src/main/resources/checkstyle/checkstyle.xml | 2 +-
.../hdfs/server/blockmanagement/HostSet.java | 25 ++--
.../hadoop/hdfs/server/namenode/JournalSet.java| 22 +-
.../hadoop/hdfs/server/protocol/RemoteEditLog.java | 11 -
.../hadoop/hdfs/server/namenode/ha/HATestUtil.java | 16 -
.../mapreduce/lib/input/TestFileInputFormat.java | 27 --
.../impl/pb/GetApplicationsRequestPBImpl.java | 12 ++
7 files changed, 44 insertions(+), 71 deletions(-)
diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
index 54a5943..e0a55f7 100644
--- a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
+++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml
@@ -123,7 +123,7 @@
+value="^com\.google\.common\.base\.(Optional|Function),
^com\.google\.common\.collect\.(ImmutableListMultimap)"/>
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
index cf7cfac..1d16cae 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HostSet.java
@@ -17,15 +17,12 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
-import com.google.common.base.Function;
-import com.google.common.base.Joiner;
+
import com.google.common.base.Preconditions;
import com.google.common.collect.HashMultimap;
-import com.google.common.collect.Iterators;
import com.google.common.collect.Multimap;
import com.google.common.collect.UnmodifiableIterator;
-import javax.annotation.Nullable;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.util.Collection;
@@ -101,14 +98,16 @@ public class HostSet implements
Iterable {
@Override
public String toString() {
StringBuilder sb = new StringBuilder("HostSet(");
-Joiner.on(",").appendTo(sb, Iterators.transform(iterator(),
-new Function() {
- @Override
- public String apply(@Nullable InetSocketAddress addr) {
-assert addr != null;
-return addr.getAddress().getHostAddress() + ":" + addr.getPort();
- }
-}));
-return sb.append(")").toString();
+Iterator iter = iterator();
+String sep = "";
+while (iter.hasNext()) {
+ InetSocketAddress addr = iter.next();
+ sb.append(sep);
+ sb.append(addr.getAddress().getHostAddress());
+ sb.append(':');
+ sb.append(addr.getPort());
+ sep = ",";
+}
+return sb.append(')').toString();
}
}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
index a5df9f5..eb3a140 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
@@ -24,8 +24,10 @@ import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
+import java.util.HashMap;
import java.util.LinkedList;
import java.util.List;
+import java.util.Map;
import java.util.PriorityQueue;
import java.util.SortedSet;
import java.util.concurrent.CopyOnWriteArrayList;
@@ -38,13 +40,9 @@ import org.apache.hadoop.hdfs.server.common.StorageInfo;
import org.apache.hadoop.hdfs.server.protocol.NamespaceInfo;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLog;
import org.apache.hadoop.hdfs.server.protocol.RemoteEditLogManifest;
-
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableListMultimap;
import com.google.common.collect.Lists;
-import com.google.common.collect.Multimaps;
impor
[hadoop] branch branch-2.10 updated: YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 0f0707f YARN-8959. TestContainerResizing fails randomly (Ahmed
Hussein via jeagles)
0f0707f is described below
commit 0f0707fb0db70e20705c9ba1be87735725f3ee6e
Author: Ahmed Hussein
AuthorDate: Wed May 6 12:48:12 2020 -0500
YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
---
.../resourcemanager/scheduler/capacity/TestContainerResizing.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
index 541539d..8161eb9 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
@@ -204,7 +204,7 @@ public class TestContainerResizing {
verifyContainerDecreased(response, containerId1, 1 * GB);
// Wait for scheduler to finish processing kill events..
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
checkUsedResource(rm1, "default", 1 * GB, null);
Assert.assertEquals(1 * GB,
@@ -605,7 +605,7 @@ public class TestContainerResizing {
// Trigger a node heartbeat..
cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
Assert.assertTrue(app.getReservedContainers().isEmpty());
@@ -714,7 +714,7 @@ public class TestContainerResizing {
am1.allocate(null, null);
// Wait for scheduler to process all events.
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 4e29738 YARN-8959. TestContainerResizing fails randomly (Ahmed
Hussein via jeagles)
4e29738 is described below
commit 4e29738f4cbb20d12ef3116b6587b6fec78a3350
Author: Ahmed Hussein
AuthorDate: Wed May 6 12:27:17 2020 -0500
YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
---
.../resourcemanager/scheduler/capacity/TestContainerResizing.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
index 307d5ae..f096680 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
@@ -205,7 +205,7 @@ public class TestContainerResizing {
verifyContainerDecreased(response, containerId1, 1 * GB);
// Wait for scheduler to finish processing kill events..
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
checkUsedResource(rm1, "default", 1 * GB, null);
Assert.assertEquals(1 * GB,
@@ -606,7 +606,7 @@ public class TestContainerResizing {
// Trigger a node heartbeat..
cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
Assert.assertTrue(app.getReservedContainers().isEmpty());
@@ -715,7 +715,7 @@ public class TestContainerResizing {
am1.allocate(null, null);
// Wait for scheduler to process all events.
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 7740b88 YARN-8959. TestContainerResizing fails randomly (Ahmed
Hussein via jeagles)
7740b88 is described below
commit 7740b88ee99fea77f4e393e4385b12afcbd0b759
Author: Ahmed Hussein
AuthorDate: Wed May 6 12:27:17 2020 -0500
YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
---
.../resourcemanager/scheduler/capacity/TestContainerResizing.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
index f9bc7c4..7379bce 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
@@ -206,7 +206,7 @@ public class TestContainerResizing {
verifyContainerDecreased(response, containerId1, 1 * GB);
// Wait for scheduler to finish processing kill events..
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
checkUsedResource(rm1, "default", 1 * GB, null);
Assert.assertEquals(1 * GB,
@@ -607,7 +607,7 @@ public class TestContainerResizing {
// Trigger a node heartbeat..
cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
Assert.assertTrue(app.getReservedContainers().isEmpty());
@@ -716,7 +716,7 @@ public class TestContainerResizing {
am1.allocate(null, null);
// Wait for scheduler to process all events.
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 7aade93 YARN-8959. TestContainerResizing fails randomly (Ahmed
Hussein via jeagles)
7aade93 is described below
commit 7aade932c7587050c75825b083ff2cc61150b201
Author: Ahmed Hussein
AuthorDate: Wed May 6 12:27:17 2020 -0500
YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
---
.../resourcemanager/scheduler/capacity/TestContainerResizing.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
index c1b1c52..e3c6e7c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
@@ -224,7 +224,7 @@ public class TestContainerResizing {
verifyContainerDecreased(response, containerId1, 1 * GB);
// Wait for scheduler to finish processing kill events..
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
checkUsedResource(rm1, "default", 1 * GB, null);
Assert.assertEquals(1 * GB,
@@ -657,7 +657,7 @@ public class TestContainerResizing {
// Trigger a node heartbeat..
cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
Assert.assertTrue(app.getReservedContainers().isEmpty());
@@ -774,7 +774,7 @@ public class TestContainerResizing {
am1.allocate(null, null);
// Wait for scheduler to process all events.
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 92e3ebb YARN-8959. TestContainerResizing fails randomly (Ahmed
Hussein via jeagles)
92e3ebb is described below
commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf
Author: Ahmed Hussein
AuthorDate: Wed May 6 12:27:17 2020 -0500
YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
---
.../resourcemanager/scheduler/capacity/TestContainerResizing.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
index c1b1c52..e3c6e7c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
@@ -224,7 +224,7 @@ public class TestContainerResizing {
verifyContainerDecreased(response, containerId1, 1 * GB);
// Wait for scheduler to finish processing kill events..
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
checkUsedResource(rm1, "default", 1 * GB, null);
Assert.assertEquals(1 * GB,
@@ -657,7 +657,7 @@ public class TestContainerResizing {
// Trigger a node heartbeat..
cs.handle(new NodeUpdateSchedulerEvent(rmNode1));
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
Assert.assertTrue(app.getReservedContainers().isEmpty());
@@ -774,7 +774,7 @@ public class TestContainerResizing {
am1.allocate(null, null);
// Wait for scheduler to process all events.
-dispatcher.waitForEventThreadToWait();
+dispatcher.await();
/* Check statuses after reservation satisfied */
// Increase request should be unreserved
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new e25ac17 YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
e25ac17 is described below
commit e25ac17e2b0a7bd860c17a0749c22a9cca2d6554
Author: Ahmed Hussein
AuthorDate: Mon May 4 10:13:51 2020 -0500
YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
---
.../scheduler/TestContainerSchedulerQueuing.java | 52 --
1 file changed, 19 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
index a31eab5..c7b1118 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
@@ -28,7 +28,6 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import com.google.common.base.Supplier;
import org.apache.hadoop.fs.UnsupportedFileSystemException;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.test.GenericTestUtils;
@@ -1256,37 +1255,24 @@ public class TestContainerSchedulerQueuing extends
BaseContainerManagerTest {
1, updateResponse.getSuccessfullyUpdatedContainers().size());
Assert.assertTrue(updateResponse.getFailedRequests().isEmpty());
-GetContainerStatusesRequest statRequest =
-
GetContainerStatusesRequest.newInstance(Collections.singletonList(cId));
-GenericTestUtils.waitFor(
-new Supplier() {
- @Override
- public Boolean get() {
-try {
- List containerStatuses = containerManager
- .getContainerStatuses(statRequest).getContainerStatuses();
- Assert.assertEquals(1, containerStatuses.size());
-
- ContainerStatus status = containerStatuses.get(0);
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
-
- return status.getExecutionType() == ExecutionType.OPPORTUNISTIC;
-} catch (Exception ex) {
- throw new RuntimeException(ex);
-}
- }
-}, 100, 1);
-List containerStatuses = containerManager
-.getContainerStatuses(statRequest).getContainerStatuses();
-Assert.assertEquals(1, containerStatuses.size());
-for (ContainerStatus status : containerStatuses) {
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
- Assert
- .assertEquals(ExecutionType.OPPORTUNISTIC,
status.getExecutionType());
-}
+final GetContainerStatusesRequest statRequest =
+GetContainerStatusesRequest.newInstance(
+Collections.singletonList(cId));
+final org.apache.hadoop.yarn.api.records.ContainerState expectedState =
+org.apache.hadoop.yarn.api.records.ContainerState.RUNNING;
+
+GenericTestUtils.waitFor(() -> {
+ List containerStatuses;
+ try {
+containerStatuses = containerManager
+.getContainerStatuses(statRequest).getContainerStatuses();
+ } catch (YarnException | IOException e) {
+return false;
+ }
+ Assert.assertEquals(1, containerStatuses.size());
+ ContainerStatus status = containerStatuses.get(0);
+ return (status.getState() == expectedState
+ && status.getExecutionType() == ExecutionType.OPPORTUNISTIC);
+}, 20, 1);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new b23a585 YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
b23a585 is described below
commit b23a585cb1dd2b4317334554187621a5084c8215
Author: Ahmed Hussein
AuthorDate: Mon May 4 10:13:51 2020 -0500
YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
---
.../scheduler/TestContainerSchedulerQueuing.java | 52 --
1 file changed, 19 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
index 258c5be..0289a2c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
@@ -28,7 +28,6 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import com.google.common.base.Supplier;
import org.apache.hadoop.fs.UnsupportedFileSystemException;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.test.GenericTestUtils;
@@ -1266,37 +1265,24 @@ public class TestContainerSchedulerQueuing extends
BaseContainerManagerTest {
1, updateResponse.getSuccessfullyUpdatedContainers().size());
Assert.assertTrue(updateResponse.getFailedRequests().isEmpty());
-GetContainerStatusesRequest statRequest =
-
GetContainerStatusesRequest.newInstance(Collections.singletonList(cId));
-GenericTestUtils.waitFor(
-new Supplier() {
- @Override
- public Boolean get() {
-try {
- List containerStatuses = containerManager
- .getContainerStatuses(statRequest).getContainerStatuses();
- Assert.assertEquals(1, containerStatuses.size());
-
- ContainerStatus status = containerStatuses.get(0);
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
-
- return status.getExecutionType() == ExecutionType.OPPORTUNISTIC;
-} catch (Exception ex) {
- throw new RuntimeException(ex);
-}
- }
-}, 100, 1);
-List containerStatuses = containerManager
-.getContainerStatuses(statRequest).getContainerStatuses();
-Assert.assertEquals(1, containerStatuses.size());
-for (ContainerStatus status : containerStatuses) {
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
- Assert
- .assertEquals(ExecutionType.OPPORTUNISTIC,
status.getExecutionType());
-}
+final GetContainerStatusesRequest statRequest =
+GetContainerStatusesRequest.newInstance(
+Collections.singletonList(cId));
+final org.apache.hadoop.yarn.api.records.ContainerState expectedState =
+org.apache.hadoop.yarn.api.records.ContainerState.RUNNING;
+
+GenericTestUtils.waitFor(() -> {
+ List containerStatuses;
+ try {
+containerStatuses = containerManager
+.getContainerStatuses(statRequest).getContainerStatuses();
+ } catch (YarnException | IOException e) {
+return false;
+ }
+ Assert.assertEquals(1, containerStatuses.size());
+ ContainerStatus status = containerStatuses.get(0);
+ return (status.getState() == expectedState
+ && status.getExecutionType() == ExecutionType.OPPORTUNISTIC);
+}, 20, 1);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 1d83500 YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
1d83500 is described below
commit 1d8350055e8f8f9fb65b188447fa9341b89c5a7d
Author: Ahmed Hussein
AuthorDate: Mon May 4 10:13:51 2020 -0500
YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
---
.../scheduler/TestContainerSchedulerQueuing.java | 52 --
1 file changed, 19 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
index 530a8e7..b21850c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
@@ -28,7 +28,6 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import com.google.common.base.Supplier;
import org.apache.hadoop.fs.UnsupportedFileSystemException;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.test.GenericTestUtils;
@@ -1291,37 +1290,24 @@ public class TestContainerSchedulerQueuing extends
BaseContainerManagerTest {
1, updateResponse.getSuccessfullyUpdatedContainers().size());
Assert.assertTrue(updateResponse.getFailedRequests().isEmpty());
-GetContainerStatusesRequest statRequest =
-
GetContainerStatusesRequest.newInstance(Collections.singletonList(cId));
-GenericTestUtils.waitFor(
-new Supplier() {
- @Override
- public Boolean get() {
-try {
- List containerStatuses = containerManager
- .getContainerStatuses(statRequest).getContainerStatuses();
- Assert.assertEquals(1, containerStatuses.size());
-
- ContainerStatus status = containerStatuses.get(0);
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
-
- return status.getExecutionType() == ExecutionType.OPPORTUNISTIC;
-} catch (Exception ex) {
- throw new RuntimeException(ex);
-}
- }
-}, 100, 1);
-List containerStatuses = containerManager
-.getContainerStatuses(statRequest).getContainerStatuses();
-Assert.assertEquals(1, containerStatuses.size());
-for (ContainerStatus status : containerStatuses) {
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
- Assert
- .assertEquals(ExecutionType.OPPORTUNISTIC,
status.getExecutionType());
-}
+final GetContainerStatusesRequest statRequest =
+GetContainerStatusesRequest.newInstance(
+Collections.singletonList(cId));
+final org.apache.hadoop.yarn.api.records.ContainerState expectedState =
+org.apache.hadoop.yarn.api.records.ContainerState.RUNNING;
+
+GenericTestUtils.waitFor(() -> {
+ List containerStatuses;
+ try {
+containerStatuses = containerManager
+.getContainerStatuses(statRequest).getContainerStatuses();
+ } catch (YarnException | IOException e) {
+return false;
+ }
+ Assert.assertEquals(1, containerStatuses.size());
+ ContainerStatus status = containerStatuses.get(0);
+ return (status.getState() == expectedState
+ && status.getExecutionType() == ExecutionType.OPPORTUNISTIC);
+}, 20, 1);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated: YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new f5081a9 YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
f5081a9 is described below
commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2
Author: Ahmed Hussein
AuthorDate: Mon May 4 10:13:51 2020 -0500
YARN-10256. Refactor
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
(Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
---
.../scheduler/TestContainerSchedulerQueuing.java | 52 --
1 file changed, 19 insertions(+), 33 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
index 530a8e7..b21850c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
@@ -28,7 +28,6 @@ import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
-import com.google.common.base.Supplier;
import org.apache.hadoop.fs.UnsupportedFileSystemException;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.test.GenericTestUtils;
@@ -1291,37 +1290,24 @@ public class TestContainerSchedulerQueuing extends
BaseContainerManagerTest {
1, updateResponse.getSuccessfullyUpdatedContainers().size());
Assert.assertTrue(updateResponse.getFailedRequests().isEmpty());
-GetContainerStatusesRequest statRequest =
-
GetContainerStatusesRequest.newInstance(Collections.singletonList(cId));
-GenericTestUtils.waitFor(
-new Supplier() {
- @Override
- public Boolean get() {
-try {
- List containerStatuses = containerManager
- .getContainerStatuses(statRequest).getContainerStatuses();
- Assert.assertEquals(1, containerStatuses.size());
-
- ContainerStatus status = containerStatuses.get(0);
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
-
- return status.getExecutionType() == ExecutionType.OPPORTUNISTIC;
-} catch (Exception ex) {
- throw new RuntimeException(ex);
-}
- }
-}, 100, 1);
-List containerStatuses = containerManager
-.getContainerStatuses(statRequest).getContainerStatuses();
-Assert.assertEquals(1, containerStatuses.size());
-for (ContainerStatus status : containerStatuses) {
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
- Assert
- .assertEquals(ExecutionType.OPPORTUNISTIC,
status.getExecutionType());
-}
+final GetContainerStatusesRequest statRequest =
+GetContainerStatusesRequest.newInstance(
+Collections.singletonList(cId));
+final org.apache.hadoop.yarn.api.records.ContainerState expectedState =
+org.apache.hadoop.yarn.api.records.ContainerState.RUNNING;
+
+GenericTestUtils.waitFor(() -> {
+ List containerStatuses;
+ try {
+containerStatuses = containerManager
+.getContainerStatuses(statRequest).getContainerStatuses();
+ } catch (YarnException | IOException e) {
+return false;
+ }
+ Assert.assertEquals(1, containerStatuses.size());
+ ContainerStatus status = containerStatuses.get(0);
+ return (status.getState() == expectedState
+ && status.getExecutionType() == ExecutionType.OPPORTUNISTIC);
+}, 20, 1);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.10 updated: YARN-10255. fix intermittent failure TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic in branch-2.10 (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 712da57 YARN-10255. fix intermittent failure
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
in branch-2.10 (Ahmed Hussein via jeagles)
712da57 is described below
commit 712da57658bb3f46f506a291cdc4d80854701a41
Author: Ahmed Hussein
AuthorDate: Thu Apr 30 17:50:55 2020 -0500
YARN-10255. fix intermittent failure
TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
in branch-2.10 (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
---
.../scheduler/TestContainerSchedulerQueuing.java | 42 +-
1 file changed, 26 insertions(+), 16 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
index 4abea57..202bd08 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
@@ -18,9 +18,11 @@
package org.apache.hadoop.yarn.server.nodemanager.containermanager.scheduler;
+import com.google.common.base.Supplier;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -29,6 +31,7 @@ import java.util.concurrent.ConcurrentMap;
import org.apache.hadoop.fs.UnsupportedFileSystemException;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.yarn.api.protocolrecords.ContainerUpdateRequest;
import org.apache.hadoop.yarn.api.protocolrecords.ContainerUpdateResponse;
import org.apache.hadoop.yarn.api.protocolrecords.GetContainerStatusesRequest;
@@ -1216,7 +1219,7 @@ public class TestContainerSchedulerQueuing extends
BaseContainerManagerTest {
@Test
public void testContainerUpdateExecTypeGuaranteedToOpportunistic()
- throws IOException, YarnException, InterruptedException {
+ throws Exception {
delayContainers = true;
containerManager.start();
// Construct the Container-id
@@ -1253,20 +1256,27 @@ public class TestContainerSchedulerQueuing extends
BaseContainerManagerTest {
1, updateResponse.getSuccessfullyUpdatedContainers().size());
Assert.assertTrue(updateResponse.getFailedRequests().isEmpty());
-//Make sure the container is running
-List statList = new ArrayList();
-statList.add(cId);
-GetContainerStatusesRequest statRequest =
-GetContainerStatusesRequest.newInstance(statList);
-List containerStatuses = containerManager
-.getContainerStatuses(statRequest).getContainerStatuses();
-Assert.assertEquals(1, containerStatuses.size());
-for (ContainerStatus status : containerStatuses) {
- Assert.assertEquals(
- org.apache.hadoop.yarn.api.records.ContainerState.RUNNING,
- status.getState());
- Assert
- .assertEquals(ExecutionType.OPPORTUNISTIC,
status.getExecutionType());
-}
+final GetContainerStatusesRequest statRequest =
+GetContainerStatusesRequest.newInstance(
+Collections.singletonList(cId));
+final org.apache.hadoop.yarn.api.records.ContainerState expectedState =
+org.apache.hadoop.yarn.api.records.ContainerState.RUNNING;
+
+GenericTestUtils.waitFor(new Supplier() {
+ @Override
+ public Boolean get() {
+List containerStatuses;
+try {
+ containerStatuses = containerManager
+ .getContainerStatuses(statRequest).getContainerStatuses();
+} catch (YarnException | IOException e) {
+ return false;
+}
+Assert.assertEquals(1, containerStatuses.size());
+ContainerStatus status = containerStatuses.get(0);
+return (status.getState() == expectedState
+&& status.getExecutionType() == ExecutionType.OPPORTUNISTIC);
+ }
+}, 20, 1);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.ap
[hadoop] branch branch-2.10 updated: MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 006b0d7 MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails
Intermittently
006b0d7 is described below
commit 006b0d70a30054a172d76b5b03ce9d52fc0ff153
Author: Ahmed Hussein
AuthorDate: Tue Jan 28 17:26:13 2020 -0600
MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
Signed-off-by: Jonathan Eagles
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 31 ---
.../v2/TestSpeculativeExecutionWithMRApp.java | 308 +
2 files changed, 133 insertions(+), 206 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index e6e971a..fd012b2 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -18,18 +18,14 @@
package org.apache.hadoop.mapreduce.v2.app;
-import com.google.common.base.Joiner;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
-import java.util.ArrayList;
-import java.util.Arrays;
import java.util.EnumSet;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileContext;
import org.apache.hadoop.fs.FileSystem;
@@ -376,33 +372,6 @@ public class MRApp extends MRAppMaster {
report.getTaskAttemptState());
}
- public void waitForState(TaskAttempt attempt,
- TaskAttemptState...finalStates) throws Exception {
-int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
-List targetStates = Arrays.asList(finalStates);
-List stateValuesList = new ArrayList<>();
-for (TaskAttemptState taState : targetStates) {
- stateValuesList.add(taState.toString());
-}
-String statesValues = Joiner.on(",").join(stateValuesList);
-while (!targetStates.contains(report.getTaskAttemptState()) &&
-timeoutSecs++ < 20) {
- System.out.println(
- "TaskAttempt " + attempt.getID().toString() + " State is : "
- + report.getTaskAttemptState()
- + " Waiting for states: " + statesValues
- + ". curent state is : " + report.getTaskAttemptState()
- + ". progress : " + report.getProgress());
- report = attempt.getReport();
- Thread.sleep(500);
-}
-System.out.println("TaskAttempt State is : "
-+ report.getTaskAttemptState());
-Assert.assertTrue("TaskAttempt state is not correct (timedout)",
-targetStates.contains(report.getTaskAttemptState()));
- }
-
public void waitForState(Task task, TaskState finalState) throws Exception {
int timeoutSecs = 0;
TaskReport report = task.getReport();
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
index d4d432b..5acee7a 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
@@ -18,16 +18,14 @@
package org.apache.hadoop.mapreduce.v2;
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
+import com.google.common.base.Supplier;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.Map;
import java.util.Random;
-import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicReference;
-
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.v2.app.speculate.LegacyTaskRuntimeEstimator;
import
org.apache.hadoop.mapreduce.v2.app.speculate.SimpleExponentialTaskRuntimeEstimator;
@@ -50,18 +48,12 @@ import
org.apache.hadoop.mapreduce
[hadoop] branch branch-3.1 updated: MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new f3ff618 MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails
Intermittently
f3ff618 is described below
commit f3ff61865a82a19c281b9e46c4b039aa2c3a1877
Author: Ahmed Hussein
AuthorDate: Tue Jan 28 10:57:33 2020 -0600
MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
Signed-off-by: Jonathan Eagles
(cherry picked from commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa)
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 27 --
.../v2/TestSpeculativeExecutionWithMRApp.java | 298 +
2 files changed, 123 insertions(+), 202 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index 9a8280b..4be80c4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -22,11 +22,8 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
-import java.util.Arrays;
import java.util.EnumSet;
-import java.util.List;
-import java.util.stream.Collectors;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileContext;
import org.apache.hadoop.fs.FileSystem;
@@ -375,30 +372,6 @@ public class MRApp extends MRAppMaster {
report.getTaskAttemptState());
}
- public void waitForState(TaskAttempt attempt,
- TaskAttemptState...finalStates) throws Exception {
-int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
-List targetStates = Arrays.asList(finalStates);
-String statesValues = targetStates.stream().map(Object::toString).collect(
-Collectors.joining(","));
-while (!targetStates.contains(report.getTaskAttemptState()) &&
-timeoutSecs++ < 20) {
- System.out.println(
- "TaskAttempt " + attempt.getID().toString() + " State is : "
- + report.getTaskAttemptState()
- + " Waiting for states: " + statesValues
- + ". curent state is : " + report.getTaskAttemptState()
- + ". progress : " + report.getProgress());
- report = attempt.getReport();
- Thread.sleep(500);
-}
-System.out.println("TaskAttempt State is : "
-+ report.getTaskAttemptState());
-Assert.assertTrue("TaskAttempt state is not correct (timedout)",
-targetStates.contains(report.getTaskAttemptState()));
- }
-
public void waitForState(Task task, TaskState finalState) throws Exception {
int timeoutSecs = 0;
TaskReport report = task.getReport();
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
index d4d432b..2163d7b 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
@@ -18,16 +18,13 @@
package org.apache.hadoop.mapreduce.v2;
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.Map;
import java.util.Random;
-import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicReference;
-
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.v2.app.speculate.LegacyTaskRuntimeEstimator;
import
org.apache.hadoop.mapreduce.v2.app.speculate.SimpleExponentialTaskRuntimeEstimator;
@@ -50,18 +47,12 @@ import
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptStatusUpdateEvent
import org.apache.hadoop.service.Service;
import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.yarn.event.EventHandler;
-import org.
[hadoop] branch branch-3.2 updated: MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 96caaf7 MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails
Intermittently
96caaf7 is described below
commit 96caaf718508211bc8148d9c5c64b9f3c77da1b9
Author: Ahmed Hussein
AuthorDate: Tue Jan 28 10:57:33 2020 -0600
MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
Signed-off-by: Jonathan Eagles
(cherry picked from commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa)
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 27 --
.../v2/TestSpeculativeExecutionWithMRApp.java | 298 +
2 files changed, 123 insertions(+), 202 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index 9a8280b..4be80c4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -22,11 +22,8 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
-import java.util.Arrays;
import java.util.EnumSet;
-import java.util.List;
-import java.util.stream.Collectors;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileContext;
import org.apache.hadoop.fs.FileSystem;
@@ -375,30 +372,6 @@ public class MRApp extends MRAppMaster {
report.getTaskAttemptState());
}
- public void waitForState(TaskAttempt attempt,
- TaskAttemptState...finalStates) throws Exception {
-int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
-List targetStates = Arrays.asList(finalStates);
-String statesValues = targetStates.stream().map(Object::toString).collect(
-Collectors.joining(","));
-while (!targetStates.contains(report.getTaskAttemptState()) &&
-timeoutSecs++ < 20) {
- System.out.println(
- "TaskAttempt " + attempt.getID().toString() + " State is : "
- + report.getTaskAttemptState()
- + " Waiting for states: " + statesValues
- + ". curent state is : " + report.getTaskAttemptState()
- + ". progress : " + report.getProgress());
- report = attempt.getReport();
- Thread.sleep(500);
-}
-System.out.println("TaskAttempt State is : "
-+ report.getTaskAttemptState());
-Assert.assertTrue("TaskAttempt state is not correct (timedout)",
-targetStates.contains(report.getTaskAttemptState()));
- }
-
public void waitForState(Task task, TaskState finalState) throws Exception {
int timeoutSecs = 0;
TaskReport report = task.getReport();
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
index d4d432b..2163d7b 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
@@ -18,16 +18,13 @@
package org.apache.hadoop.mapreduce.v2;
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.Map;
import java.util.Random;
-import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicReference;
-
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.v2.app.speculate.LegacyTaskRuntimeEstimator;
import
org.apache.hadoop.mapreduce.v2.app.speculate.SimpleExponentialTaskRuntimeEstimator;
@@ -50,18 +47,12 @@ import
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptStatusUpdateEvent
import org.apache.hadoop.service.Service;
import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.yarn.event.EventHandler;
-import org.
[hadoop] branch trunk updated: MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 0825153 MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails
Intermittently
0825153 is described below
commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa
Author: Ahmed Hussein
AuthorDate: Tue Jan 28 10:57:33 2020 -0600
MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
Signed-off-by: Jonathan Eagles
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 27 --
.../v2/TestSpeculativeExecutionWithMRApp.java | 298 +
2 files changed, 123 insertions(+), 202 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index 9a8280b..4be80c4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -22,11 +22,8 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.InetSocketAddress;
-import java.util.Arrays;
import java.util.EnumSet;
-import java.util.List;
-import java.util.stream.Collectors;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileContext;
import org.apache.hadoop.fs.FileSystem;
@@ -375,30 +372,6 @@ public class MRApp extends MRAppMaster {
report.getTaskAttemptState());
}
- public void waitForState(TaskAttempt attempt,
- TaskAttemptState...finalStates) throws Exception {
-int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
-List targetStates = Arrays.asList(finalStates);
-String statesValues = targetStates.stream().map(Object::toString).collect(
-Collectors.joining(","));
-while (!targetStates.contains(report.getTaskAttemptState()) &&
-timeoutSecs++ < 20) {
- System.out.println(
- "TaskAttempt " + attempt.getID().toString() + " State is : "
- + report.getTaskAttemptState()
- + " Waiting for states: " + statesValues
- + ". curent state is : " + report.getTaskAttemptState()
- + ". progress : " + report.getProgress());
- report = attempt.getReport();
- Thread.sleep(500);
-}
-System.out.println("TaskAttempt State is : "
-+ report.getTaskAttemptState());
-Assert.assertTrue("TaskAttempt state is not correct (timedout)",
-targetStates.contains(report.getTaskAttemptState()));
- }
-
public void waitForState(Task task, TaskState finalState) throws Exception {
int timeoutSecs = 0;
TaskReport report = task.getReport();
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
index d4d432b..2163d7b 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
@@ -18,16 +18,13 @@
package org.apache.hadoop.mapreduce.v2;
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
import java.util.Map;
import java.util.Random;
-import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicReference;
-
import org.apache.hadoop.mapreduce.MRJobConfig;
import org.apache.hadoop.mapreduce.v2.app.speculate.LegacyTaskRuntimeEstimator;
import
org.apache.hadoop.mapreduce.v2.app.speculate.SimpleExponentialTaskRuntimeEstimator;
@@ -50,18 +47,12 @@ import
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptStatusUpdateEvent
import org.apache.hadoop.service.Service;
import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.yarn.event.EventHandler;
-import org.apache.hadoop.yarn.util.Clock;
import org.apache.hadoop.yarn.util.ControlledClock;
-import org.apache.hadoop.yarn
[hadoop] branch branch-2.10 updated: MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 8e22b80 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
8e22b80 is described below
commit 8e22b80b5fba5a62a87f78a385c384a79db380f4
Author: Ahmed Hussein
AuthorDate: Tue Jan 28 09:13:54 2020 -0600
MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
Signed-off-by: Jonathan Eagles
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 90 +-
1 file changed, 35 insertions(+), 55 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index d195228..e6e971a 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -27,8 +27,8 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.EnumSet;
-import org.apache.commons.logging.Log;
-import org.apache.commons.logging.LogFactory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileContext;
@@ -106,13 +106,16 @@ import org.apache.hadoop.yarn.util.SystemClock;
import org.junit.Assert;
+
/**
* Mock MRAppMaster. Doesn't start RPC servers.
* No threads are started except of the event Dispatcher thread.
*/
@SuppressWarnings("unchecked")
public class MRApp extends MRAppMaster {
- private static final Log LOG = LogFactory.getLog(MRApp.class);
+ private static final Logger LOG = LoggerFactory.getLogger(MRApp.class);
+ private static final int WAIT_FOR_STATE_CNT = 200;
+ private static final int WAIT_FOR_STATE_INTERVAL= 50;
/**
* The available resource of each container allocated.
@@ -322,13 +325,11 @@ public class MRApp extends MRAppMaster {
JobStateInternal finalState) throws Exception {
int timeoutSecs = 0;
JobStateInternal iState = job.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Job Internal State is : " + iState
- + " Waiting for Internal state : " + finalState);
- Thread.sleep(500);
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = job.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Job {} Internal State is : {}", job.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -336,17 +337,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskImpl task,
TaskStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskReport report = task.getReport();
TaskStateInternal iState = task.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Task Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = task.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = task.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Task {} Internal State is : {}", task.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -354,17 +350,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskAttemptImpl attempt,
TaskAttemptStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
TaskAttemptStateInternal iState = attempt.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("TaskAttempt Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = attempt.getRepor
[hadoop] branch branch-3.1 updated: MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 1f315c2 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
1f315c2 is described below
commit 1f315c28d96982da273b7da84f5aed283f45bc7b
Author: Ahmed Hussein
AuthorDate: Mon Jan 27 15:50:13 2020 -0600
MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 83 --
1 file changed, 31 insertions(+), 52 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index 70ea18a..9a8280b 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -113,6 +113,8 @@ import org.slf4j.LoggerFactory;
@SuppressWarnings("unchecked")
public class MRApp extends MRAppMaster {
private static final Logger LOG = LoggerFactory.getLogger(MRApp.class);
+ private static final int WAIT_FOR_STATE_CNT = 200;
+ private static final int WAIT_FOR_STATE_INTERVAL= 50;
/**
* The available resource of each container allocated.
@@ -322,13 +324,11 @@ public class MRApp extends MRAppMaster {
JobStateInternal finalState) throws Exception {
int timeoutSecs = 0;
JobStateInternal iState = job.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Job Internal State is : " + iState
- + " Waiting for Internal state : " + finalState);
- Thread.sleep(500);
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = job.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Job {} Internal State is : {}", job.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -336,17 +336,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskImpl task,
TaskStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskReport report = task.getReport();
TaskStateInternal iState = task.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Task Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = task.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = task.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Task {} Internal State is : {}", task.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -354,17 +349,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskAttemptImpl attempt,
TaskAttemptStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
TaskAttemptStateInternal iState = attempt.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("TaskAttempt Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = attempt.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = attempt.getInternalState();
}
-System.out.println("TaskAttempt Internal State is : " + iState);
+LOG.info("TaskAttempt {} Internal State is : {}", attempt.getID(), iState);
Assert.assertEquals("TaskAttempt Internal state is not correct (timedout)",
finalState, iState);
}
@@ -374,17 +364,12 @@ public class MRApp
[hadoop] branch branch-3.2 updated: MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 6538e57 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
6538e57 is described below
commit 6538e57c41e5af374649e601013dadec01e915e3
Author: Ahmed Hussein
AuthorDate: Mon Jan 27 15:50:13 2020 -0600
MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 83 --
1 file changed, 31 insertions(+), 52 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index 70ea18a..9a8280b 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -113,6 +113,8 @@ import org.slf4j.LoggerFactory;
@SuppressWarnings("unchecked")
public class MRApp extends MRAppMaster {
private static final Logger LOG = LoggerFactory.getLogger(MRApp.class);
+ private static final int WAIT_FOR_STATE_CNT = 200;
+ private static final int WAIT_FOR_STATE_INTERVAL= 50;
/**
* The available resource of each container allocated.
@@ -322,13 +324,11 @@ public class MRApp extends MRAppMaster {
JobStateInternal finalState) throws Exception {
int timeoutSecs = 0;
JobStateInternal iState = job.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Job Internal State is : " + iState
- + " Waiting for Internal state : " + finalState);
- Thread.sleep(500);
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = job.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Job {} Internal State is : {}", job.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -336,17 +336,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskImpl task,
TaskStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskReport report = task.getReport();
TaskStateInternal iState = task.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Task Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = task.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = task.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Task {} Internal State is : {}", task.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -354,17 +349,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskAttemptImpl attempt,
TaskAttemptStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
TaskAttemptStateInternal iState = attempt.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("TaskAttempt Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = attempt.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = attempt.getInternalState();
}
-System.out.println("TaskAttempt Internal State is : " + iState);
+LOG.info("TaskAttempt {} Internal State is : {}", attempt.getID(), iState);
Assert.assertEquals("TaskAttempt Internal state is not correct (timedout)",
finalState, iState);
}
@@ -374,17 +364,12 @@ public class MRApp
[hadoop] branch trunk updated: MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3f01c48 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
3f01c48 is described below
commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b
Author: Ahmed Hussein
AuthorDate: Mon Jan 27 15:50:13 2020 -0600
MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
Signed-off-by: Jonathan Eagles
---
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 83 --
1 file changed, 31 insertions(+), 52 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
index 70ea18a..9a8280b 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
@@ -113,6 +113,8 @@ import org.slf4j.LoggerFactory;
@SuppressWarnings("unchecked")
public class MRApp extends MRAppMaster {
private static final Logger LOG = LoggerFactory.getLogger(MRApp.class);
+ private static final int WAIT_FOR_STATE_CNT = 200;
+ private static final int WAIT_FOR_STATE_INTERVAL= 50;
/**
* The available resource of each container allocated.
@@ -322,13 +324,11 @@ public class MRApp extends MRAppMaster {
JobStateInternal finalState) throws Exception {
int timeoutSecs = 0;
JobStateInternal iState = job.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Job Internal State is : " + iState
- + " Waiting for Internal state : " + finalState);
- Thread.sleep(500);
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = job.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Job {} Internal State is : {}", job.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -336,17 +336,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskImpl task,
TaskStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskReport report = task.getReport();
TaskStateInternal iState = task.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("Task Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = task.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = task.getInternalState();
}
-System.out.println("Task Internal State is : " + iState);
+LOG.info("Task {} Internal State is : {}", task.getID(), iState);
Assert.assertEquals("Task Internal state is not correct (timedout)",
finalState, iState);
}
@@ -354,17 +349,12 @@ public class MRApp extends MRAppMaster {
public void waitForInternalState(TaskAttemptImpl attempt,
TaskAttemptStateInternal finalState) throws Exception {
int timeoutSecs = 0;
-TaskAttemptReport report = attempt.getReport();
TaskAttemptStateInternal iState = attempt.getInternalState();
-while (!finalState.equals(iState) && timeoutSecs++ < 20) {
- System.out.println("TaskAttempt Internal State is : " + iState
- + " Waiting for Internal state : " + finalState + " progress : "
- + report.getProgress());
- Thread.sleep(500);
- report = attempt.getReport();
+while (!finalState.equals(iState) && timeoutSecs++ < WAIT_FOR_STATE_CNT) {
+ Thread.sleep(WAIT_FOR_STATE_INTERVAL);
iState = attempt.getInternalState();
}
-System.out.println("TaskAttempt Internal State is : " + iState);
+LOG.info("TaskAttempt {} Internal State is : {}", attempt.getID(), iState);
Assert.assertEquals("TaskAttempt Internal state is not correct (timedout)",
finalState, iState);
}
@@ -374,17 +364,12 @@ public class MRApp extends MRAppMaster {
int timeoutSecs
[hadoop] branch branch-2.9 updated: HADOOP-16749. Configuration parsing of CDATA values are blank
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new bc53074 HADOOP-16749. Configuration parsing of CDATA values are blank
bc53074 is described below
commit bc530743b3cd3ef6876aa0a19607ebe8c0097e52
Author: Daryn Sharp
AuthorDate: Fri Jan 10 10:48:45 2020 -0600
HADOOP-16749. Configuration parsing of CDATA values are blank
Signed-off-by: Jonathan Eagles
(cherry picked from commit 23a77050e08a8d57581da43fa8d6cd11ad8eaa0e)
---
.../java/org/apache/hadoop/conf/Configuration.java | 1 +
.../org/apache/hadoop/conf/TestConfiguration.java | 38 ++
2 files changed, 39 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 9ee1baf..6601b23 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -2889,6 +2889,7 @@ public class Configuration implements
Iterable>,
break;
case XMLStreamConstants.CHARACTERS:
+case XMLStreamConstants.CDATA:
if (parseToken) {
char[] text = reader.getTextCharacters();
token.append(text, reader.getTextStart(), reader.getTextLength());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
index 6fcff58..398f0aa 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
@@ -63,6 +63,7 @@ import org.apache.log4j.Logger;
import org.apache.log4j.spi.LoggingEvent;
import org.hamcrest.CoreMatchers;
import org.junit.Assert;
+import org.junit.Test;
import org.mockito.Mockito;
import static org.apache.hadoop.util.PlatformName.IBM_JAVA;
@@ -2095,4 +2096,41 @@ public class TestConfiguration extends TestCase {
TestConfiguration.class.getName()
});
}
+
+ @Test
+ public void testCDATA() throws IOException {
+String xml = new String(
+"" +
+ "" +
+"cdata" +
+"" +
+ "\n" +
+ "" +
+"cdata-multiple" +
+" and " +
+ "\n" +
+ "" +
+"cdata-multiline" +
+"" +
+ "\n" +
+ "" +
+"cdata-whitespace" +
+" prefix \nsuffix " +
+ "\n" +
+"");
+Configuration conf = checkCDATA(xml.getBytes());
+ByteArrayOutputStream os = new ByteArrayOutputStream();
+conf.writeXml(os);
+checkCDATA(os.toByteArray());
+ }
+
+ private static Configuration checkCDATA(byte[] bytes) {
+Configuration conf = new Configuration(false);
+conf.addResource(new ByteArrayInputStream(bytes));
+assertEquals(">cdata", conf.get("cdata"));
+assertEquals(">cdata1 and >cdata2", conf.get("cdata-multiple"));
+assertEquals(">cdata\nmultiline<>", conf.get("cdata-multiline"));
+assertEquals(" prefix >cdata\nsuffix ", conf.get("cdata-whitespace"));
+return conf;
+ }
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.10 updated: HADOOP-16749. Configuration parsing of CDATA values are blank
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 23a7705 HADOOP-16749. Configuration parsing of CDATA values are blank
23a7705 is described below
commit 23a77050e08a8d57581da43fa8d6cd11ad8eaa0e
Author: Daryn Sharp
AuthorDate: Fri Jan 10 10:48:45 2020 -0600
HADOOP-16749. Configuration parsing of CDATA values are blank
Signed-off-by: Jonathan Eagles
---
.../java/org/apache/hadoop/conf/Configuration.java | 1 +
.../org/apache/hadoop/conf/TestConfiguration.java | 38 ++
2 files changed, 39 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index bb1a02e..de9893b 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -2925,6 +2925,7 @@ public class Configuration implements
Iterable>,
break;
case XMLStreamConstants.CHARACTERS:
+case XMLStreamConstants.CDATA:
if (parseToken) {
char[] text = reader.getTextCharacters();
token.append(text, reader.getTextStart(), reader.getTextLength());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
index f5daf2c..d491a52 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
@@ -63,6 +63,7 @@ import org.apache.log4j.Logger;
import org.apache.log4j.spi.LoggingEvent;
import org.hamcrest.CoreMatchers;
import org.junit.Assert;
+import org.junit.Test;
import org.mockito.Mockito;
import static org.apache.hadoop.util.PlatformName.IBM_JAVA;
@@ -2104,4 +2105,41 @@ public class TestConfiguration extends TestCase {
TestConfiguration.class.getName()
});
}
+
+ @Test
+ public void testCDATA() throws IOException {
+String xml = new String(
+"" +
+ "" +
+"cdata" +
+"" +
+ "\n" +
+ "" +
+"cdata-multiple" +
+" and " +
+ "\n" +
+ "" +
+"cdata-multiline" +
+"" +
+ "\n" +
+ "" +
+"cdata-whitespace" +
+" prefix \nsuffix " +
+ "\n" +
+"");
+Configuration conf = checkCDATA(xml.getBytes());
+ByteArrayOutputStream os = new ByteArrayOutputStream();
+conf.writeXml(os);
+checkCDATA(os.toByteArray());
+ }
+
+ private static Configuration checkCDATA(byte[] bytes) {
+Configuration conf = new Configuration(false);
+conf.addResource(new ByteArrayInputStream(bytes));
+assertEquals(">cdata", conf.get("cdata"));
+assertEquals(">cdata1 and >cdata2", conf.get("cdata-multiple"));
+assertEquals(">cdata\nmultiline<>", conf.get("cdata-multiline"));
+assertEquals(" prefix >cdata\nsuffix ", conf.get("cdata-whitespace"));
+return conf;
+ }
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: HADOOP-16749. Configuration parsing of CDATA values are blank
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new da45684 HADOOP-16749. Configuration parsing of CDATA values are blank
da45684 is described below
commit da45684185a24969963b169c13b322b29921dc82
Author: Daryn Sharp
AuthorDate: Fri Jan 10 10:31:39 2020 -0600
HADOOP-16749. Configuration parsing of CDATA values are blank
Signed-off-by: Jonathan Eagles
(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)
---
.../java/org/apache/hadoop/conf/Configuration.java | 1 +
.../org/apache/hadoop/conf/TestConfiguration.java | 37 ++
2 files changed, 38 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 72beb92..e0111ef 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -3322,6 +3322,7 @@ public class Configuration implements
Iterable>,
handleStartElement();
break;
case XMLStreamConstants.CHARACTERS:
+ case XMLStreamConstants.CDATA:
if (parseToken) {
char[] text = reader.getTextCharacters();
token.append(text, reader.getTextStart(), reader.getTextLength());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
index 0703fe7..5dc5c40 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
@@ -2573,4 +2573,41 @@ public class TestConfiguration {
// Thread 1
config.get("secondParse");
}
+
+ @Test
+ public void testCDATA() throws IOException {
+String xml = new String(
+"" +
+ "" +
+"cdata" +
+"" +
+ "\n" +
+ "" +
+"cdata-multiple" +
+" and " +
+ "\n" +
+ "" +
+"cdata-multiline" +
+"" +
+ "\n" +
+ "" +
+"cdata-whitespace" +
+" prefix \nsuffix " +
+ "\n" +
+"");
+Configuration conf = checkCDATA(xml.getBytes());
+ByteArrayOutputStream os = new ByteArrayOutputStream();
+conf.writeXml(os);
+checkCDATA(os.toByteArray());
+ }
+
+ private static Configuration checkCDATA(byte[] bytes) {
+Configuration conf = new Configuration(false);
+conf.addResource(new ByteArrayInputStream(bytes));
+assertEquals(">cdata", conf.get("cdata"));
+assertEquals(">cdata1 and >cdata2", conf.get("cdata-multiple"));
+assertEquals(">cdata\nmultiline<>", conf.get("cdata-multiline"));
+assertEquals(" prefix >cdata\nsuffix ", conf.get("cdata-whitespace"));
+return conf;
+ }
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: HADOOP-16749. Configuration parsing of CDATA values are blank
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 57cb180 HADOOP-16749. Configuration parsing of CDATA values are blank
57cb180 is described below
commit 57cb18026776f1be7172e523c68842edac9aaee9
Author: Daryn Sharp
AuthorDate: Fri Jan 10 10:31:39 2020 -0600
HADOOP-16749. Configuration parsing of CDATA values are blank
Signed-off-by: Jonathan Eagles
(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)
---
.../java/org/apache/hadoop/conf/Configuration.java | 1 +
.../org/apache/hadoop/conf/TestConfiguration.java | 37 ++
2 files changed, 38 insertions(+)
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index c6ceea5..873fd4a 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -3322,6 +3322,7 @@ public class Configuration implements
Iterable>,
handleStartElement();
break;
case XMLStreamConstants.CHARACTERS:
+ case XMLStreamConstants.CDATA:
if (parseToken) {
char[] text = reader.getTextCharacters();
token.append(text, reader.getTextStart(), reader.getTextLength());
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
index 64beb7b..aebefe0 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
@@ -2553,4 +2553,41 @@ public class TestConfiguration {
// Thread 1
config.get("secondParse");
}
+
+ @Test
+ public void testCDATA() throws IOException {
+String xml = new String(
+"" +
+ "" +
+"cdata" +
+"" +
+ "\n" +
+ "" +
+"cdata-multiple" +
+" and " +
+ "\n" +
+ "" +
+"cdata-multiline" +
+"" +
+ "\n" +
+ "" +
+"cdata-whitespace" +
+" prefix \nsuffix " +
+ "\n" +
+"");
+Configuration conf = checkCDATA(xml.getBytes());
+ByteArrayOutputStream os = new ByteArrayOutputStream();
+conf.writeXml(os);
+checkCDATA(os.toByteArray());
+ }
+
+ private static Configuration checkCDATA(byte[] bytes) {
+Configuration conf = new Configuration(false);
+conf.addResource(new ByteArrayInputStream(bytes));
+assertEquals(">cdata", conf.get("cdata"));
+assertEquals(">cdata1 and >cdata2", conf.get("cdata-multiple"));
+assertEquals(">cdata\nmultiline<>", conf.get("cdata-multiline"));
+assertEquals(" prefix >cdata\nsuffix ", conf.get("cdata-whitespace"));
+return conf;
+ }
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-2.10 updated: MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 203f55f MAPREDUCE-7252. Handling 0 progress in SimpleExponential task
runtime estimator
203f55f is described below
commit 203f55fccc96cabddc777438c9c9eb6ed3e88e0a
Author: Ahmed Hussein
AuthorDate: Thu Jan 9 09:52:26 2020 -0600
MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime
estimator
Signed-off-by: Jonathan Eagles
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 28 +++--
.../SimpleExponentialTaskRuntimeEstimator.java | 67 +++
.../forecast/SimpleExponentialSmoothing.java | 131 +
.../v2/app/speculate/forecast/package-info.java| 20
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 46 +++-
.../v2/TestSpeculativeExecutionWithMRApp.java | 116 --
6 files changed, 312 insertions(+), 96 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index 9f1c122..036eb45 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -18,6 +18,11 @@
package org.apache.hadoop.mapreduce.v2.app.speculate;
public class DataStatistics {
+
+ /**
+ * factor used to calculate confidence interval within 95%.
+ */
+ private static final double DEFAULT_CI_FACTOR = 1.96;
private int count = 0;
private double sum = 0;
private double sumSquares = 0;
@@ -25,25 +30,26 @@ public class DataStatistics {
public DataStatistics() {
}
- public DataStatistics(double initNum) {
+ public DataStatistics(final double initNum) {
this.count = 1;
this.sum = initNum;
this.sumSquares = initNum * initNum;
}
- public synchronized void add(double newNum) {
+ public synchronized void add(final double newNum) {
this.count++;
this.sum += newNum;
this.sumSquares += newNum * newNum;
}
- public synchronized void updateStatistics(double old, double update) {
- this.sum += update - old;
- this.sumSquares += (update * update) - (old * old);
+ public synchronized void updateStatistics(final double old,
+ final double update) {
+this.sum += update - old;
+this.sumSquares += (update * update) - (old * old);
}
public synchronized double mean() {
-return count == 0 ? 0.0 : sum/count;
+return count == 0 ? 0.0 : sum / count;
}
public synchronized double var() {
@@ -52,14 +58,14 @@ public class DataStatistics {
return 0.0;
}
double mean = mean();
-return Math.max((sumSquares/count) - mean * mean, 0.0d);
+return Math.max((sumSquares / count) - mean * mean, 0.0d);
}
public synchronized double std() {
return Math.sqrt(this.var());
}
- public synchronized double outlier(float sigma) {
+ public synchronized double outlier(final float sigma) {
if (count != 0.0) {
return mean() + std() * sigma;
}
@@ -78,10 +84,12 @@ public class DataStatistics {
* @return the mean value adding 95% confidence interval
*/
public synchronized double meanCI() {
-if (count <= 1) return 0.0;
+if (count <= 1) {
+ return 0.0;
+}
double currMean = mean();
double currStd = std();
-return currMean + (1.96 * currStd / Math.sqrt(count));
+return currMean + (DEFAULT_CI_FACTOR * currStd / Math.sqrt(count));
}
public String toString() {
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
index f244b20..2838916 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
@@ -33,7 +33,22 @@ import
org.apache.hadoop.mapreduce.v2.app.speculate.forecast.SimpleExponentialSm
* A task Runtime Estimator
[hadoop] branch branch-3.1 updated: MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 6ad2f66 MAPREDUCE-7252. Handling 0 progress in SimpleExponential task
runtime estimator
6ad2f66 is described below
commit 6ad2f662eb4995637c537fd1f23de9e246c023a1
Author: Ahmed Hussein
AuthorDate: Wed Jan 8 11:08:13 2020 -0600
MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime
estimator
Signed-off-by: Jonathan Eagles
(cherry picked from commit cdd6efd3ab6917e30b4c5c7b261f61838901bb37)
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 28 +++--
.../SimpleExponentialTaskRuntimeEstimator.java | 67 +++
.../forecast/SimpleExponentialSmoothing.java | 131 +
.../v2/app/speculate/forecast/package-info.java| 20
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 42 ++-
.../v2/TestSpeculativeExecutionWithMRApp.java | 116 --
6 files changed, 308 insertions(+), 96 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index 9f1c122..036eb45 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -18,6 +18,11 @@
package org.apache.hadoop.mapreduce.v2.app.speculate;
public class DataStatistics {
+
+ /**
+ * factor used to calculate confidence interval within 95%.
+ */
+ private static final double DEFAULT_CI_FACTOR = 1.96;
private int count = 0;
private double sum = 0;
private double sumSquares = 0;
@@ -25,25 +30,26 @@ public class DataStatistics {
public DataStatistics() {
}
- public DataStatistics(double initNum) {
+ public DataStatistics(final double initNum) {
this.count = 1;
this.sum = initNum;
this.sumSquares = initNum * initNum;
}
- public synchronized void add(double newNum) {
+ public synchronized void add(final double newNum) {
this.count++;
this.sum += newNum;
this.sumSquares += newNum * newNum;
}
- public synchronized void updateStatistics(double old, double update) {
- this.sum += update - old;
- this.sumSquares += (update * update) - (old * old);
+ public synchronized void updateStatistics(final double old,
+ final double update) {
+this.sum += update - old;
+this.sumSquares += (update * update) - (old * old);
}
public synchronized double mean() {
-return count == 0 ? 0.0 : sum/count;
+return count == 0 ? 0.0 : sum / count;
}
public synchronized double var() {
@@ -52,14 +58,14 @@ public class DataStatistics {
return 0.0;
}
double mean = mean();
-return Math.max((sumSquares/count) - mean * mean, 0.0d);
+return Math.max((sumSquares / count) - mean * mean, 0.0d);
}
public synchronized double std() {
return Math.sqrt(this.var());
}
- public synchronized double outlier(float sigma) {
+ public synchronized double outlier(final float sigma) {
if (count != 0.0) {
return mean() + std() * sigma;
}
@@ -78,10 +84,12 @@ public class DataStatistics {
* @return the mean value adding 95% confidence interval
*/
public synchronized double meanCI() {
-if (count <= 1) return 0.0;
+if (count <= 1) {
+ return 0.0;
+}
double currMean = mean();
double currStd = std();
-return currMean + (1.96 * currStd / Math.sqrt(count));
+return currMean + (DEFAULT_CI_FACTOR * currStd / Math.sqrt(count));
}
public String toString() {
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
index f244b20..2838916 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
@@ -33,7 +33,22 @@ import
org.apache.hadoop.mapred
[hadoop] branch branch-3.2 updated: MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new ee7c56c MAPREDUCE-7252. Handling 0 progress in SimpleExponential task
runtime estimator
ee7c56c is described below
commit ee7c56c6de4e89d3fab5140afcc1237e9e6fc63e
Author: Ahmed Hussein
AuthorDate: Wed Jan 8 11:08:13 2020 -0600
MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime
estimator
Signed-off-by: Jonathan Eagles
(cherry picked from commit cdd6efd3ab6917e30b4c5c7b261f61838901bb37)
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 28 +++--
.../SimpleExponentialTaskRuntimeEstimator.java | 67 +++
.../forecast/SimpleExponentialSmoothing.java | 131 +
.../v2/app/speculate/forecast/package-info.java| 20
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 42 ++-
.../v2/TestSpeculativeExecutionWithMRApp.java | 116 --
6 files changed, 308 insertions(+), 96 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index 9f1c122..036eb45 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -18,6 +18,11 @@
package org.apache.hadoop.mapreduce.v2.app.speculate;
public class DataStatistics {
+
+ /**
+ * factor used to calculate confidence interval within 95%.
+ */
+ private static final double DEFAULT_CI_FACTOR = 1.96;
private int count = 0;
private double sum = 0;
private double sumSquares = 0;
@@ -25,25 +30,26 @@ public class DataStatistics {
public DataStatistics() {
}
- public DataStatistics(double initNum) {
+ public DataStatistics(final double initNum) {
this.count = 1;
this.sum = initNum;
this.sumSquares = initNum * initNum;
}
- public synchronized void add(double newNum) {
+ public synchronized void add(final double newNum) {
this.count++;
this.sum += newNum;
this.sumSquares += newNum * newNum;
}
- public synchronized void updateStatistics(double old, double update) {
- this.sum += update - old;
- this.sumSquares += (update * update) - (old * old);
+ public synchronized void updateStatistics(final double old,
+ final double update) {
+this.sum += update - old;
+this.sumSquares += (update * update) - (old * old);
}
public synchronized double mean() {
-return count == 0 ? 0.0 : sum/count;
+return count == 0 ? 0.0 : sum / count;
}
public synchronized double var() {
@@ -52,14 +58,14 @@ public class DataStatistics {
return 0.0;
}
double mean = mean();
-return Math.max((sumSquares/count) - mean * mean, 0.0d);
+return Math.max((sumSquares / count) - mean * mean, 0.0d);
}
public synchronized double std() {
return Math.sqrt(this.var());
}
- public synchronized double outlier(float sigma) {
+ public synchronized double outlier(final float sigma) {
if (count != 0.0) {
return mean() + std() * sigma;
}
@@ -78,10 +84,12 @@ public class DataStatistics {
* @return the mean value adding 95% confidence interval
*/
public synchronized double meanCI() {
-if (count <= 1) return 0.0;
+if (count <= 1) {
+ return 0.0;
+}
double currMean = mean();
double currStd = std();
-return currMean + (1.96 * currStd / Math.sqrt(count));
+return currMean + (DEFAULT_CI_FACTOR * currStd / Math.sqrt(count));
}
public String toString() {
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
index f244b20..2838916 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
@@ -33,7 +33,22 @@ import
org.apache.hadoop.mapred
[hadoop] branch trunk updated: MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new cdd6efd MAPREDUCE-7252. Handling 0 progress in SimpleExponential task
runtime estimator
cdd6efd is described below
commit cdd6efd3ab6917e30b4c5c7b261f61838901bb37
Author: Ahmed Hussein
AuthorDate: Wed Jan 8 11:08:13 2020 -0600
MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime
estimator
Signed-off-by: Jonathan Eagles
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 28 +++--
.../SimpleExponentialTaskRuntimeEstimator.java | 67 +++
.../forecast/SimpleExponentialSmoothing.java | 131 +
.../v2/app/speculate/forecast/package-info.java| 20
.../org/apache/hadoop/mapreduce/v2/app/MRApp.java | 42 ++-
.../v2/TestSpeculativeExecutionWithMRApp.java | 116 --
6 files changed, 308 insertions(+), 96 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index 9f1c122..036eb45 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -18,6 +18,11 @@
package org.apache.hadoop.mapreduce.v2.app.speculate;
public class DataStatistics {
+
+ /**
+ * factor used to calculate confidence interval within 95%.
+ */
+ private static final double DEFAULT_CI_FACTOR = 1.96;
private int count = 0;
private double sum = 0;
private double sumSquares = 0;
@@ -25,25 +30,26 @@ public class DataStatistics {
public DataStatistics() {
}
- public DataStatistics(double initNum) {
+ public DataStatistics(final double initNum) {
this.count = 1;
this.sum = initNum;
this.sumSquares = initNum * initNum;
}
- public synchronized void add(double newNum) {
+ public synchronized void add(final double newNum) {
this.count++;
this.sum += newNum;
this.sumSquares += newNum * newNum;
}
- public synchronized void updateStatistics(double old, double update) {
- this.sum += update - old;
- this.sumSquares += (update * update) - (old * old);
+ public synchronized void updateStatistics(final double old,
+ final double update) {
+this.sum += update - old;
+this.sumSquares += (update * update) - (old * old);
}
public synchronized double mean() {
-return count == 0 ? 0.0 : sum/count;
+return count == 0 ? 0.0 : sum / count;
}
public synchronized double var() {
@@ -52,14 +58,14 @@ public class DataStatistics {
return 0.0;
}
double mean = mean();
-return Math.max((sumSquares/count) - mean * mean, 0.0d);
+return Math.max((sumSquares / count) - mean * mean, 0.0d);
}
public synchronized double std() {
return Math.sqrt(this.var());
}
- public synchronized double outlier(float sigma) {
+ public synchronized double outlier(final float sigma) {
if (count != 0.0) {
return mean() + std() * sigma;
}
@@ -78,10 +84,12 @@ public class DataStatistics {
* @return the mean value adding 95% confidence interval
*/
public synchronized double meanCI() {
-if (count <= 1) return 0.0;
+if (count <= 1) {
+ return 0.0;
+}
double currMean = mean();
double currStd = std();
-return currMean + (1.96 * currStd / Math.sqrt(count));
+return currMean + (DEFAULT_CI_FACTOR * currStd / Math.sqrt(count));
}
public String toString() {
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
index f244b20..2838916 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
@@ -33,7 +33,22 @@ import
org.apache.hadoop.mapreduce.v2.app.speculate.forecast.SimpleExponentialSm
* A task Runtime Estimator based on expon
[hadoop] branch branch-2.10 updated: MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new b4c633a MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein
via jeagles)
b4c633a is described below
commit b4c633a9818463067a092ee194a7fc4acf81c683
Author: Ahmed Hussein
AuthorDate: Tue Nov 5 15:48:27 2019 -0600
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit c59b1b66a5264c3164ac6c9356b51de9b5349c66)
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 18 +-
.../v2/app/speculate/DefaultSpeculator.java| 3 +-
.../v2/app/speculate/NullTaskRuntimesEngine.java | 5 +
.../SimpleExponentialTaskRuntimeEstimator.java | 170
.../v2/app/speculate/StartEndTimesBase.java| 8 +-
.../v2/app/speculate/TaskRuntimeEstimator.java | 13 +
.../forecast/SimpleExponentialSmoothing.java | 196 +
.../mapreduce/v2/app/TestRuntimeEstimators.java| 8 +
.../forecast/TestSimpleExponentialForecast.java| 120 +++
.../org/apache/hadoop/mapreduce/MRJobConfig.java | 31 +
.../mapreduce/v2/TestSpeculativeExecOnCluster.java | 935 +
.../v2/TestSpeculativeExecutionWithMRApp.java | 46 +-
12 files changed, 1546 insertions(+), 7 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index cfaffaf..9f1c122 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -71,8 +71,22 @@ public class DataStatistics {
return count;
}
+ /**
+ * calculates the mean value within 95% ConfidenceInterval.
+ * 1.96 is standard for 95 %
+ *
+ * @return the mean value adding 95% confidence interval
+ */
+ public synchronized double meanCI() {
+if (count <= 1) return 0.0;
+double currMean = mean();
+double currStd = std();
+return currMean + (1.96 * currStd / Math.sqrt(count));
+ }
+
public String toString() {
-return "DataStatistics: count is " + count + ", sum is " + sum +
-", sumSquares is " + sumSquares + " mean is " + mean() + " std() is " +
std();
+return "DataStatistics: count is " + count + ", sum is " + sum
++ ", sumSquares is " + sumSquares + " mean is " + mean()
++ " std() is " + std() + ", meanCI() is " + meanCI();
}
}
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
index c6bc441..4ab8f71 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
@@ -414,7 +414,8 @@ public class DefaultSpeculator extends AbstractService
implements
if (estimatedRunTime == data.getEstimatedRunTime()
&& progress == data.getProgress()) {
// Previous stats are same as same stats
-if (data.notHeartbeatedInAWhile(now)) {
+if (data.notHeartbeatedInAWhile(now)
+|| estimator.hasStagnatedProgress(runningTaskAttemptID, now)) {
// Stats have stagnated for a while, simulate heart-beat.
TaskAttemptStatus taskAttemptStatus = new TaskAttemptStatus();
taskAttemptStatus.id = runningTaskAttemptID;
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/NullTaskRuntimesEngine.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/NullTaskRuntimesEngine.java
index 7211ff4..eb52bc7 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/
[hadoop] branch branch-2 updated: MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new c59b1b6 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein
via jeagles)
c59b1b6 is described below
commit c59b1b66a5264c3164ac6c9356b51de9b5349c66
Author: Ahmed Hussein
AuthorDate: Tue Nov 5 15:48:27 2019 -0600
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 18 +-
.../v2/app/speculate/DefaultSpeculator.java| 3 +-
.../v2/app/speculate/NullTaskRuntimesEngine.java | 5 +
.../SimpleExponentialTaskRuntimeEstimator.java | 170
.../v2/app/speculate/StartEndTimesBase.java| 8 +-
.../v2/app/speculate/TaskRuntimeEstimator.java | 13 +
.../forecast/SimpleExponentialSmoothing.java | 196 +
.../mapreduce/v2/app/TestRuntimeEstimators.java| 8 +
.../forecast/TestSimpleExponentialForecast.java| 120 +++
.../org/apache/hadoop/mapreduce/MRJobConfig.java | 31 +
.../mapreduce/v2/TestSpeculativeExecOnCluster.java | 935 +
.../v2/TestSpeculativeExecutionWithMRApp.java | 46 +-
12 files changed, 1546 insertions(+), 7 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index cfaffaf..9f1c122 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -71,8 +71,22 @@ public class DataStatistics {
return count;
}
+ /**
+ * calculates the mean value within 95% ConfidenceInterval.
+ * 1.96 is standard for 95 %
+ *
+ * @return the mean value adding 95% confidence interval
+ */
+ public synchronized double meanCI() {
+if (count <= 1) return 0.0;
+double currMean = mean();
+double currStd = std();
+return currMean + (1.96 * currStd / Math.sqrt(count));
+ }
+
public String toString() {
-return "DataStatistics: count is " + count + ", sum is " + sum +
-", sumSquares is " + sumSquares + " mean is " + mean() + " std() is " +
std();
+return "DataStatistics: count is " + count + ", sum is " + sum
++ ", sumSquares is " + sumSquares + " mean is " + mean()
++ " std() is " + std() + ", meanCI() is " + meanCI();
}
}
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
index c6bc441..4ab8f71 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
@@ -414,7 +414,8 @@ public class DefaultSpeculator extends AbstractService
implements
if (estimatedRunTime == data.getEstimatedRunTime()
&& progress == data.getProgress()) {
// Previous stats are same as same stats
-if (data.notHeartbeatedInAWhile(now)) {
+if (data.notHeartbeatedInAWhile(now)
+|| estimator.hasStagnatedProgress(runningTaskAttemptID, now)) {
// Stats have stagnated for a while, simulate heart-beat.
TaskAttemptStatus taskAttemptStatus = new TaskAttemptStatus();
taskAttemptStatus.id = runningTaskAttemptID;
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/NullTaskRuntimesEngine.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/NullTaskRuntimesEngine.java
index 7211ff4..eb52bc7 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/NullTaskRuntimesEngine.java
+++
b/hadoop-mapredu
[hadoop] branch branch-3.1 updated: MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new cbd5016 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein
via jeagles)
cbd5016 is described below
commit cbd501636c1bf048109864def87118f266802480
Author: Ahmed Hussein
AuthorDate: Tue Nov 5 14:55:20 2019 -0600
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 18 +-
.../v2/app/speculate/DefaultSpeculator.java| 3 +-
.../SimpleExponentialTaskRuntimeEstimator.java | 170
.../v2/app/speculate/StartEndTimesBase.java| 3 +-
.../v2/app/speculate/TaskRuntimeEstimator.java | 15 +
.../forecast/SimpleExponentialSmoothing.java | 196 +
.../mapreduce/v2/app/TestRuntimeEstimators.java| 8 +
.../forecast/TestSimpleExponentialForecast.java| 120 +++
.../org/apache/hadoop/mapreduce/MRJobConfig.java | 31 +
.../mapreduce/v2/TestSpeculativeExecOnCluster.java | 935 +
.../v2/TestSpeculativeExecutionWithMRApp.java | 46 +-
11 files changed, 1538 insertions(+), 7 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index cfaffaf..9f1c122 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -71,8 +71,22 @@ public class DataStatistics {
return count;
}
+ /**
+ * calculates the mean value within 95% ConfidenceInterval.
+ * 1.96 is standard for 95 %
+ *
+ * @return the mean value adding 95% confidence interval
+ */
+ public synchronized double meanCI() {
+if (count <= 1) return 0.0;
+double currMean = mean();
+double currStd = std();
+return currMean + (1.96 * currStd / Math.sqrt(count));
+ }
+
public String toString() {
-return "DataStatistics: count is " + count + ", sum is " + sum +
-", sumSquares is " + sumSquares + " mean is " + mean() + " std() is " +
std();
+return "DataStatistics: count is " + count + ", sum is " + sum
++ ", sumSquares is " + sumSquares + " mean is " + mean()
++ " std() is " + std() + ", meanCI() is " + meanCI();
}
}
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
index 6573687..211bdc4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
@@ -416,7 +416,8 @@ public class DefaultSpeculator extends AbstractService
implements
if (estimatedRunTime == data.getEstimatedRunTime()
&& progress == data.getProgress()) {
// Previous stats are same as same stats
-if (data.notHeartbeatedInAWhile(now)) {
+if (data.notHeartbeatedInAWhile(now)
+|| estimator.hasStagnatedProgress(runningTaskAttemptID, now)) {
// Stats have stagnated for a while, simulate heart-beat.
TaskAttemptStatus taskAttemptStatus = new TaskAttemptStatus();
taskAttemptStatus.id = runningTaskAttemptID;
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
new file mode 100644
index 000..f244b20
--- /dev/null
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/specul
[hadoop] branch branch-3.2 updated: MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 56988e8 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein
via jeagles)
56988e8 is described below
commit 56988e88f7b4cc67b9aa03cfc71aaa5f05c0d859
Author: Ahmed Hussein
AuthorDate: Tue Nov 5 14:55:20 2019 -0600
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 18 +-
.../v2/app/speculate/DefaultSpeculator.java| 3 +-
.../SimpleExponentialTaskRuntimeEstimator.java | 170
.../v2/app/speculate/StartEndTimesBase.java| 3 +-
.../v2/app/speculate/TaskRuntimeEstimator.java | 15 +
.../forecast/SimpleExponentialSmoothing.java | 196 +
.../mapreduce/v2/app/TestRuntimeEstimators.java| 8 +
.../forecast/TestSimpleExponentialForecast.java| 120 +++
.../org/apache/hadoop/mapreduce/MRJobConfig.java | 31 +
.../mapreduce/v2/TestSpeculativeExecOnCluster.java | 935 +
.../v2/TestSpeculativeExecutionWithMRApp.java | 46 +-
11 files changed, 1538 insertions(+), 7 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index cfaffaf..9f1c122 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -71,8 +71,22 @@ public class DataStatistics {
return count;
}
+ /**
+ * calculates the mean value within 95% ConfidenceInterval.
+ * 1.96 is standard for 95 %
+ *
+ * @return the mean value adding 95% confidence interval
+ */
+ public synchronized double meanCI() {
+if (count <= 1) return 0.0;
+double currMean = mean();
+double currStd = std();
+return currMean + (1.96 * currStd / Math.sqrt(count));
+ }
+
public String toString() {
-return "DataStatistics: count is " + count + ", sum is " + sum +
-", sumSquares is " + sumSquares + " mean is " + mean() + " std() is " +
std();
+return "DataStatistics: count is " + count + ", sum is " + sum
++ ", sumSquares is " + sumSquares + " mean is " + mean()
++ " std() is " + std() + ", meanCI() is " + meanCI();
}
}
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
index 6573687..211bdc4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
@@ -416,7 +416,8 @@ public class DefaultSpeculator extends AbstractService
implements
if (estimatedRunTime == data.getEstimatedRunTime()
&& progress == data.getProgress()) {
// Previous stats are same as same stats
-if (data.notHeartbeatedInAWhile(now)) {
+if (data.notHeartbeatedInAWhile(now)
+|| estimator.hasStagnatedProgress(runningTaskAttemptID, now)) {
// Stats have stagnated for a while, simulate heart-beat.
TaskAttemptStatus taskAttemptStatus = new TaskAttemptStatus();
taskAttemptStatus.id = runningTaskAttemptID;
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
new file mode 100644
index 000..f244b20
--- /dev/null
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/specul
[hadoop] branch branch-3.2 updated: Revert "YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph."
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 254e18d Revert "YARN-9949. Add missing queue configs for root queue
in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph."
254e18d is described below
commit 254e18dcaf293608880964daa37a15e5e30ea949
Author: Jonathan Eagles
AuthorDate: Tue Nov 5 15:10:01 2019 -0600
Revert "YARN-9949. Add missing queue configs for root queue in
RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph."
This reverts commit 11c763c22055fea367b19b338a3d8067f9386ba4.
---
.../capacity/CapacitySchedulerConfiguration.java | 8
.../policy/AbstractComparatorOrderingPolicy.java | 3 --
.../scheduler/policy/FairOrderingPolicy.java | 6 ---
.../scheduler/policy/FifoOrderingPolicy.java | 6 ---
.../policy/FifoOrderingPolicyForPendingApps.java | 6 ---
.../FifoOrderingPolicyWithExclusivePartitions.java | 7 ---
.../scheduler/policy/OrderingPolicy.java | 6 ---
.../webapp/CapacitySchedulerPage.java | 2 +-
.../webapp/dao/CapacitySchedulerInfo.java | 51 --
.../webapp/dao/CapacitySchedulerLeafQueueInfo.java | 10 +
.../webapp/TestRMWebServicesCapacitySched.java | 42 +-
.../TestRMWebServicesForCSWithPartitions.java | 2 +-
12 files changed, 4 insertions(+), 145 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
index f98a129..760636e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
@@ -50,7 +50,6 @@ import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.MultiNo
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.MultiNodePolicySpec;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.FairOrderingPolicy;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.FifoOrderingPolicy;
-import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.FifoOrderingPolicyForPendingApps;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.FifoOrderingPolicyWithExclusivePartitions;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.OrderingPolicy;
import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.policy.SchedulableEntity;
@@ -164,9 +163,6 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
public static final String FIFO_WITH_PARTITIONS_APP_ORDERING_POLICY
= "fifo-with-partitions";
- public static final String FIFO_FOR_PENDING_APPS
- = "fifo-for-pending-apps";
-
public static final String DEFAULT_APP_ORDERING_POLICY =
FIFO_APP_ORDERING_POLICY;
@@ -577,10 +573,6 @@ public class CapacitySchedulerConfiguration extends
ReservationSchedulerConfigur
if (policyType.trim().equals(FIFO_WITH_PARTITIONS_APP_ORDERING_POLICY)) {
policyType = FifoOrderingPolicyWithExclusivePartitions.class.getName();
}
-if (policyType.trim().equals(FIFO_FOR_PENDING_APPS)) {
- policyType = FifoOrderingPolicyForPendingApps.class.getName();
-}
-
try {
orderingPolicy = (OrderingPolicy)
Class.forName(policyType).newInstance();
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java
index b5e870b..4ee4450 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/Abstrac
[hadoop] branch trunk updated: MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new ed302f1 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein
via jeagles)
ed302f1 is described below
commit ed302f1fed6d124d682486d24dae958946dba9be
Author: Ahmed Hussein
AuthorDate: Tue Nov 5 14:55:20 2019 -0600
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles
---
.../mapreduce/v2/app/speculate/DataStatistics.java | 18 +-
.../v2/app/speculate/DefaultSpeculator.java| 3 +-
.../SimpleExponentialTaskRuntimeEstimator.java | 170
.../v2/app/speculate/StartEndTimesBase.java| 3 +-
.../v2/app/speculate/TaskRuntimeEstimator.java | 15 +
.../forecast/SimpleExponentialSmoothing.java | 196 +
.../mapreduce/v2/app/TestRuntimeEstimators.java| 8 +
.../forecast/TestSimpleExponentialForecast.java| 120 +++
.../org/apache/hadoop/mapreduce/MRJobConfig.java | 31 +
.../mapreduce/v2/TestSpeculativeExecOnCluster.java | 935 +
.../v2/TestSpeculativeExecutionWithMRApp.java | 46 +-
11 files changed, 1538 insertions(+), 7 deletions(-)
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
index cfaffaf..9f1c122 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DataStatistics.java
@@ -71,8 +71,22 @@ public class DataStatistics {
return count;
}
+ /**
+ * calculates the mean value within 95% ConfidenceInterval.
+ * 1.96 is standard for 95 %
+ *
+ * @return the mean value adding 95% confidence interval
+ */
+ public synchronized double meanCI() {
+if (count <= 1) return 0.0;
+double currMean = mean();
+double currStd = std();
+return currMean + (1.96 * currStd / Math.sqrt(count));
+ }
+
public String toString() {
-return "DataStatistics: count is " + count + ", sum is " + sum +
-", sumSquares is " + sumSquares + " mean is " + mean() + " std() is " +
std();
+return "DataStatistics: count is " + count + ", sum is " + sum
++ ", sumSquares is " + sumSquares + " mean is " + mean()
++ " std() is " + std() + ", meanCI() is " + meanCI();
}
}
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
index 6573687..211bdc4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java
@@ -416,7 +416,8 @@ public class DefaultSpeculator extends AbstractService
implements
if (estimatedRunTime == data.getEstimatedRunTime()
&& progress == data.getProgress()) {
// Previous stats are same as same stats
-if (data.notHeartbeatedInAWhile(now)) {
+if (data.notHeartbeatedInAWhile(now)
+|| estimator.hasStagnatedProgress(runningTaskAttemptID, now)) {
// Stats have stagnated for a while, simulate heart-beat.
TaskAttemptStatus taskAttemptStatus = new TaskAttemptStatus();
taskAttemptStatus.id = runningTaskAttemptID;
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
new file mode 100644
index 000..f244b20
--- /dev/null
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/SimpleExponentialTaskRuntimeEstimator.java
@@ -0,0 +1,170 @@
+/**
+ * Li
[hadoop] branch branch-2 updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2 by this push:
new dc4d7f6 YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
dc4d7f6 is described below
commit dc4d7f69cec3f47af5be6f24cee1439081a4845c
Author: Ahmed Hussein
AuthorDate: Wed May 29 14:19:20 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 9d63fc7ed61ca6c3d901fe9984a7cca98ce07ea9)
---
.../resourcemanager/scheduler/SchedulerApplicationAttempt.java | 5 +++--
.../resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java | 4 +++-
.../server/resourcemanager/scheduler/capacity/TestLeafQueue.java | 9 +
3 files changed, 15 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 05dc834..71abdf0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1037,9 +1037,10 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
-if (queueCapacityPerc != 0) {
- queueUsagePerc = calc.divide(cluster, usedResourceClone,
+queueUsagePerc = calc.divide(cluster, usedResourceClone,
Resources.multiply(cluster, queueCapacityPerc)) * 100;
+if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+ queueUsagePerc = 0.0f;
}
clusterUsagePerc = calc.divide(cluster, usedResourceClone, cluster)
* 100;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
index e7bb027..6f37623 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
@@ -1031,7 +1031,9 @@ public class FiCaSchedulerApp extends
SchedulerApplicationAttempt {
Resources.multiply(totalPartitionRes,
queueAbsMaxCapPerPartition))
* 100;
}
-report.setQueueUsagePercentage(queueUsagePerc);
+if (!(Float.isNaN(queueUsagePerc) ||
Float.isInfinite(queueUsagePerc))) {
+ report.setQueueUsagePercentage(queueUsagePerc);
+}
}
return report;
} finally {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
index 115373b..3b76fb2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
@@ -3967,6 +3967,15 @@ public class TestLeafQueue {
0.01f);
assertEquals(15.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// test that queueUsagePercentage returns neither NaN nor Infinite
+AbstractCSQueue zeroQueue = createQueue("test2.2", null,
+Float.MIN_VALUE, Float.MIN_VALUE);
+app = new FiCaSchedulerApp(appA
[hadoop] branch branch-2.9 updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.9
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.9 by this push:
new 97fc94d YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
97fc94d is described below
commit 97fc94d3899ba7a4fcff42836190a2dd14191024
Author: Ahmed Hussein
AuthorDate: Wed May 29 14:19:20 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit 9d63fc7ed61ca6c3d901fe9984a7cca98ce07ea9)
---
.../resourcemanager/scheduler/SchedulerApplicationAttempt.java | 5 +++--
.../resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java | 4 +++-
.../server/resourcemanager/scheduler/capacity/TestLeafQueue.java | 9 +
3 files changed, 15 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 59fd814..67e565b 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1046,9 +1046,10 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
-if (queueCapacityPerc != 0) {
- queueUsagePerc = calc.divide(cluster, usedResourceClone,
+queueUsagePerc = calc.divide(cluster, usedResourceClone,
Resources.multiply(cluster, queueCapacityPerc)) * 100;
+if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+ queueUsagePerc = 0.0f;
}
clusterUsagePerc = calc.divide(cluster, usedResourceClone, cluster)
* 100;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
index e3f6cb5..9562eea 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
@@ -1021,7 +1021,9 @@ public class FiCaSchedulerApp extends
SchedulerApplicationAttempt {
Resources.multiply(totalPartitionRes,
queueAbsMaxCapPerPartition))
* 100;
}
-report.setQueueUsagePercentage(queueUsagePerc);
+if (!(Float.isNaN(queueUsagePerc) ||
Float.isInfinite(queueUsagePerc))) {
+ report.setQueueUsagePercentage(queueUsagePerc);
+}
}
return report;
} finally {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
index 115373b..3b76fb2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
@@ -3967,6 +3967,15 @@ public class TestLeafQueue {
0.01f);
assertEquals(15.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// test that queueUsagePercentage returns neither NaN nor Infinite
+AbstractCSQueue zeroQueue = createQueue("test2.2", null,
+Float.MIN_VALUE, Float.MIN_VALUE);
+app = new FiCaSchedulerApp(appA
[hadoop] branch branch-3.0 updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 9d63fc7 YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
9d63fc7 is described below
commit 9d63fc7ed61ca6c3d901fe9984a7cca98ce07ea9
Author: Ahmed Hussein
AuthorDate: Wed May 29 14:19:20 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
---
.../resourcemanager/scheduler/SchedulerApplicationAttempt.java | 5 +++--
.../resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java | 4 +++-
.../server/resourcemanager/scheduler/capacity/TestLeafQueue.java | 9 +
3 files changed, 15 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 7f912c2..7de2976 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1049,9 +1049,10 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
-if (queueCapacityPerc != 0) {
- queueUsagePerc = calc.divide(cluster, usedResourceClone,
+queueUsagePerc = calc.divide(cluster, usedResourceClone,
Resources.multiply(cluster, queueCapacityPerc)) * 100;
+if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+ queueUsagePerc = 0.0f;
}
clusterUsagePerc =
calc.divide(cluster, usedResourceClone, cluster) * 100;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
index 697b9d6..d63fd53 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
@@ -1019,7 +1019,9 @@ public class FiCaSchedulerApp extends
SchedulerApplicationAttempt {
Resources.multiply(totalPartitionRes,
queueAbsMaxCapPerPartition))
* 100;
}
-report.setQueueUsagePercentage(queueUsagePerc);
+if (!(Float.isNaN(queueUsagePerc) ||
Float.isInfinite(queueUsagePerc))) {
+ report.setQueueUsagePercentage(queueUsagePerc);
+}
}
return report;
} finally {
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
index 9f7919e..51d5589 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
@@ -3959,6 +3959,15 @@ public class TestLeafQueue {
0.01f);
assertEquals(15.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// test that queueUsagePercentage returns neither NaN nor Infinite
+AbstractCSQueue zeroQueue = createQueue("test2.2", null,
+Float.MIN_VALUE, Float.MIN_VALUE);
+app = new FiCaSchedulerApp(appAttId, user, zeroQueue,
+qChild.getAbstractUsersManager(),
[hadoop] branch branch-2.8 updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-2.8
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.8 by this push:
new 404a7e7 YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
404a7e7 is described below
commit 404a7e75a2155d4c0e3e4e95869e886ea7ab8b82
Author: Ahmed Hussein
AuthorDate: Wed May 29 13:41:03 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
---
.../scheduler/SchedulerApplicationAttempt.java | 3 +++
.../scheduler/common/fica/FiCaSchedulerApp.java | 4 +++-
.../scheduler/TestSchedulerApplicationAttempt.java | 13 +
.../resourcemanager/scheduler/capacity/TestLeafQueue.java | 8
4 files changed, 27 insertions(+), 1 deletion(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 7a957eb..45e485e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -717,6 +717,9 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
queueUsagePerc =
calc.divide(cluster, usedResourceClone, Resources.multiply(cluster,
queue.getQueueInfo(false, false).getCapacity())) * 100;
+ if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+queueUsagePerc = 0.0f;
+ }
clusterUsagePerc = calc.divide(cluster, usedResourceClone, cluster) *
100;
}
return ApplicationResourceUsageReport.newInstance(liveContainers.size(),
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
index 9663719..b3a7fbc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
@@ -657,7 +657,9 @@ public class FiCaSchedulerApp extends
SchedulerApplicationAttempt {
calc.divide(totalPartitionRes, report.getUsedResources(),
Resources.multiply(totalPartitionRes,
queueAbsMaxCapPerPartition)) * 100;
- report.setQueueUsagePercentage(queueUsagePerc);
+ if (!(Float.isInfinite(queueUsagePerc) || Float.isNaN(queueUsagePerc))) {
+report.setQueueUsagePercentage(queueUsagePerc);
+ }
}
return report;
}
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
index cf70bd8..9cae52c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
@@ -267,6 +267,19 @@ public class TestSchedulerApplicationAttempt {
0.01f);
assertEquals(60.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// Test NaN values and Infinite values
+queue = createQueue("test3", null, Float.MIN_VALUE);
+app = new SchedulerApplicationAttempt(appAttId, user, queue,
+queue.getActiveUsersManager(), rmContext);
+
+// Resource request
+app.attemptResourceUsa
[hadoop] branch branch-3.1 updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new f2202f7 YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
f2202f7 is described below
commit f2202f799090b729a90b244200c4fe5d081ab919
Author: Ahmed Hussein
AuthorDate: Wed May 29 11:24:08 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit abf76ac371e4611c8eb371736b433e3d89c9d2ae)
---
.../resourcemanager/scheduler/SchedulerApplicationAttempt.java | 7 ---
.../scheduler/TestSchedulerApplicationAttempt.java | 2 +-
.../resourcemanager/scheduler/capacity/TestLeafQueue.java | 10 ++
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 767df6d..3e78220 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1124,9 +1124,10 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
-if (queueCapacityPerc != 0) {
- queueUsagePerc = calc.divide(cluster, usedResourceClone,
- Resources.multiply(cluster, queueCapacityPerc)) * 100;
+queueUsagePerc = calc.divide(cluster, usedResourceClone,
+Resources.multiply(cluster, queueCapacityPerc)) * 100;
+if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+ queueUsagePerc = 0.0f;
}
clusterUsagePerc =
calc.divide(cluster, usedResourceClone, cluster) * 100;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
index c110b1c..0d6ff16 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
@@ -271,7 +271,7 @@ public class TestSchedulerApplicationAttempt {
assertEquals(60.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
-queue = createQueue("test3", null, 0.0f);
+queue = createQueue("test3", null, Float.MIN_VALUE);
app = new SchedulerApplicationAttempt(appAttId, user, queue,
queue.getAbstractUsersManager(), rmContext);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
index 04bb791..15a5eb2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
@@ -4037,6 +4037,16 @@ public class TestLeafQueue {
0.01f);
assertEquals(15.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// test that queueUsagePercentage returns neither NaN nor Infinite
+AbstractCSQueue zeroQueue = createQueue("test2.2", null,
+Float.MIN_VALUE, Float.MIN_VALUE,
+
[hadoop] branch branch-3.2 updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 777f734 YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
777f734 is described below
commit 777f7345ef93c9ef1497b9cee55f17211f8de959
Author: Ahmed Hussein
AuthorDate: Wed May 29 11:24:08 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
(cherry picked from commit abf76ac371e4611c8eb371736b433e3d89c9d2ae)
---
.../resourcemanager/scheduler/SchedulerApplicationAttempt.java | 7 ---
.../scheduler/TestSchedulerApplicationAttempt.java | 2 +-
.../resourcemanager/scheduler/capacity/TestLeafQueue.java | 10 ++
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index f9df2b8..c777db0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1124,9 +1124,10 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
-if (queueCapacityPerc != 0) {
- queueUsagePerc = calc.divide(cluster, usedResourceClone,
- Resources.multiply(cluster, queueCapacityPerc)) * 100;
+queueUsagePerc = calc.divide(cluster, usedResourceClone,
+Resources.multiply(cluster, queueCapacityPerc)) * 100;
+if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+ queueUsagePerc = 0.0f;
}
clusterUsagePerc =
calc.divide(cluster, usedResourceClone, cluster) * 100;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
index b7a143b..c3fe5f5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
@@ -276,7 +276,7 @@ public class TestSchedulerApplicationAttempt {
assertEquals(60.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
-queue = createQueue("test3", null, 0.0f);
+queue = createQueue("test3", null, Float.MIN_VALUE);
app = new SchedulerApplicationAttempt(appAttId, user, queue,
queue.getAbstractUsersManager(), rmContext);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
index 04bb791..15a5eb2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
@@ -4037,6 +4037,16 @@ public class TestLeafQueue {
0.01f);
assertEquals(15.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// test that queueUsagePercentage returns neither NaN nor Infinite
+AbstractCSQueue zeroQueue = createQueue("test2.2", null,
+Float.MIN_VALUE, Float.MIN_VALUE,
+
[hadoop] branch trunk updated: YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
This is an automated email from the ASF dual-hosted git repository.
jeagles pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new abf76ac YARN-9563. Resource report REST API could return NaN or Inf
(Ahmed Hussein via jeagles)
abf76ac is described below
commit abf76ac371e4611c8eb371736b433e3d89c9d2ae
Author: Ahmed Hussein
AuthorDate: Wed May 29 11:24:08 2019 -0500
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein
via jeagles)
Signed-off-by: Jonathan Eagles
---
.../resourcemanager/scheduler/SchedulerApplicationAttempt.java | 7 ---
.../scheduler/TestSchedulerApplicationAttempt.java | 2 +-
.../resourcemanager/scheduler/capacity/TestLeafQueue.java | 10 ++
3 files changed, 15 insertions(+), 4 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
index 2e8a7c1..cc7f585 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java
@@ -1125,9 +1125,10 @@ public class SchedulerApplicationAttempt implements
SchedulableEntity {
if (!calc.isInvalidDivisor(cluster)) {
float queueCapacityPerc = queue.getQueueInfo(false, false)
.getCapacity();
-if (queueCapacityPerc != 0) {
- queueUsagePerc = calc.divide(cluster, usedResourceClone,
- Resources.multiply(cluster, queueCapacityPerc)) * 100;
+queueUsagePerc = calc.divide(cluster, usedResourceClone,
+Resources.multiply(cluster, queueCapacityPerc)) * 100;
+if (Float.isNaN(queueUsagePerc) || Float.isInfinite(queueUsagePerc)) {
+ queueUsagePerc = 0.0f;
}
clusterUsagePerc =
calc.divide(cluster, usedResourceClone, cluster) * 100;
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
index b7a143b..c3fe5f5 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
@@ -276,7 +276,7 @@ public class TestSchedulerApplicationAttempt {
assertEquals(60.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
-queue = createQueue("test3", null, 0.0f);
+queue = createQueue("test3", null, Float.MIN_VALUE);
app = new SchedulerApplicationAttempt(appAttId, user, queue,
queue.getAbstractUsersManager(), rmContext);
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
index e21f9e3..29fe142 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
@@ -4039,6 +4039,16 @@ public class TestLeafQueue {
0.01f);
assertEquals(15.0f,
app.getResourceUsageReport().getClusterUsagePercentage(), 0.01f);
+
+// test that queueUsagePercentage returns neither NaN nor Infinite
+AbstractCSQueue zeroQueue = createQueue("test2.2", null,
+Float.MIN_VALUE, Float.MIN_VALUE,
+Resources.multiply(res, Float.MIN_VALUE));
+app = new FiCaSchedule
hadoop git commit: HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.8 94f4b5b9f -> 23150c29f
HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles)
(cherry picked from commit be06504a125d0f8ca222047ebd2dfd414bad3427)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/23150c29
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/23150c29
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/23150c29
Branch: refs/heads/branch-2.8
Commit: 23150c29ff2d5aa2c2de9720a9ac99046d29c0b8
Parents: 94f4b5b
Author: Jonathan Eagles
Authored: Wed Oct 10 10:37:20 2018 -0500
Committer: Jonathan Eagles
Committed: Wed Oct 10 10:37:58 2018 -0500
--
.../apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java| 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/23150c29/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
--
diff --git
a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
index 31fac9f..2d154b3 100644
---
a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
+++
b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
@@ -19,7 +19,7 @@ package org.apache.hadoop.crypto.key.kms.server;
import org.apache.hadoop.classification.InterfaceAudience;
import org.codehaus.jackson.map.ObjectMapper;
-
+import org.codehaus.jackson.map.ObjectWriter;
import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
@@ -44,6 +44,8 @@ import java.util.Map;
@Produces(MediaType.APPLICATION_JSON)
@InterfaceAudience.Private
public class KMSJSONWriter implements MessageBodyWriter {
+ private static final ObjectWriter WRITER =
+ new ObjectMapper().writerWithDefaultPrettyPrinter();
@Override
public boolean isWriteable(Class aClass, Type type,
@@ -65,8 +67,7 @@ public class KMSJSONWriter implements
MessageBodyWriter {
OutputStream outputStream) throws IOException, WebApplicationException {
Writer writer = new OutputStreamWriter(outputStream, Charset
.forName("UTF-8"));
-ObjectMapper jsonMapper = new ObjectMapper();
-jsonMapper.writerWithDefaultPrettyPrinter().writeValue(writer, obj);
+WRITER.writeValue(writer, obj);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.9 e75191a62 -> be06504a1
HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/be06504a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/be06504a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/be06504a
Branch: refs/heads/branch-2.9
Commit: be06504a125d0f8ca222047ebd2dfd414bad3427
Parents: e75191a
Author: Jonathan Eagles
Authored: Wed Oct 10 10:37:20 2018 -0500
Committer: Jonathan Eagles
Committed: Wed Oct 10 10:37:20 2018 -0500
--
.../apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java| 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/be06504a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
--
diff --git
a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
index 31fac9f..2d154b3 100644
---
a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
+++
b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
@@ -19,7 +19,7 @@ package org.apache.hadoop.crypto.key.kms.server;
import org.apache.hadoop.classification.InterfaceAudience;
import org.codehaus.jackson.map.ObjectMapper;
-
+import org.codehaus.jackson.map.ObjectWriter;
import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
@@ -44,6 +44,8 @@ import java.util.Map;
@Produces(MediaType.APPLICATION_JSON)
@InterfaceAudience.Private
public class KMSJSONWriter implements MessageBodyWriter {
+ private static final ObjectWriter WRITER =
+ new ObjectMapper().writerWithDefaultPrettyPrinter();
@Override
public boolean isWriteable(Class aClass, Type type,
@@ -65,8 +67,7 @@ public class KMSJSONWriter implements
MessageBodyWriter {
OutputStream outputStream) throws IOException, WebApplicationException {
Writer writer = new OutputStreamWriter(outputStream, Charset
.forName("UTF-8"));
-ObjectMapper jsonMapper = new ObjectMapper();
-jsonMapper.writerWithDefaultPrettyPrinter().writeValue(writer, obj);
+WRITER.writeValue(writer, obj);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-3.0 b170de8be -> 2b6d8c1e2
HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2b6d8c1e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2b6d8c1e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2b6d8c1e
Branch: refs/heads/branch-3.0
Commit: 2b6d8c1e295f9f0304271b6e1658b28d5ca4c3bf
Parents: b170de8
Author: Jonathan Eagles
Authored: Wed Oct 10 10:32:43 2018 -0500
Committer: Jonathan Eagles
Committed: Wed Oct 10 10:32:43 2018 -0500
--
.../apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java| 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2b6d8c1e/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
--
diff --git
a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
index f826572..4f002aa 100644
---
a/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
+++
b/hadoop-common-project/hadoop-kms/src/main/java/org/apache/hadoop/crypto/key/kms/server/KMSJSONWriter.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.crypto.key.kms.server;
import com.fasterxml.jackson.databind.ObjectMapper;
-
+import com.fasterxml.jackson.databind.ObjectWriter;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.http.JettyUtils;
@@ -46,6 +46,8 @@ import java.util.Map;
@Produces(MediaType.APPLICATION_JSON + "; " + JettyUtils.UTF_8)
@InterfaceAudience.Private
public class KMSJSONWriter implements MessageBodyWriter {
+ private static final ObjectWriter WRITER =
+ new ObjectMapper().writerWithDefaultPrettyPrinter();
@Override
public boolean isWriteable(Class aClass, Type type,
@@ -67,8 +69,7 @@ public class KMSJSONWriter implements
MessageBodyWriter {
OutputStream outputStream) throws IOException, WebApplicationException {
Writer writer = new OutputStreamWriter(outputStream, Charset
.forName("UTF-8"));
-ObjectMapper jsonMapper = new ObjectMapper();
-jsonMapper.writerWithDefaultPrettyPrinter().writeValue(writer, obj);
+WRITER.writeValue(writer, obj);
}
}
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-15550. Avoid static initialization of ObjectMappers
Repository: hadoop
Updated Branches:
refs/heads/branch-3.1 be0ce74c7 -> 89d448102
HADOOP-15550. Avoid static initialization of ObjectMappers
(cherry picked from commit 7a3c6e9c3cd9ffdc71946fd12f5c3d59718c4939)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/89d44810
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/89d44810
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/89d44810
Branch: refs/heads/branch-3.1
Commit: 89d4481020ed50d54d750de204c86d2b3b76a939
Parents: be0ce74
Author: Todd Lipcon
Authored: Mon Jun 25 15:36:45 2018 -0700
Committer: Jonathan Eagles
Committed: Tue Oct 9 13:50:00 2018 -0500
--
.../crypto/key/kms/KMSClientProvider.java | 7 ++
.../web/DelegationTokenAuthenticator.java | 8 ++-
.../apache/hadoop/util/HttpExceptionUtils.java | 12 ++
.../apache/hadoop/util/JsonSerialization.java | 24
.../crypto/key/kms/server/KMSJSONWriter.java| 6 ++---
.../hadoop/hdfs/web/WebHdfsFileSystem.java | 7 ++
...onfRefreshTokenBasedAccessTokenProvider.java | 8 +++
.../CredentialBasedAccessTokenProvider.java | 8 +++
.../apache/hadoop/mapreduce/JobSubmitter.java | 8 +++
.../hadoop/fs/azure/security/JsonUtils.java | 4 ++--
10 files changed, 45 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/89d44810/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
index edbf897..7b46075 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
@@ -42,6 +42,7 @@ import org.apache.hadoop.security.token.TokenRenewer;
import
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenIdentifier;
import
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL;
import org.apache.hadoop.util.HttpExceptionUtils;
+import org.apache.hadoop.util.JsonSerialization;
import org.apache.hadoop.util.KMSUtil;
import org.apache.http.client.utils.URIBuilder;
import org.slf4j.Logger;
@@ -79,7 +80,6 @@ import
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension;
import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.CryptoExtension;
import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.ObjectWriter;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
@@ -132,9 +132,6 @@ public class KMSClientProvider extends KeyProvider
implements CryptoExtension,
private final ValueQueue encKeyVersionQueue;
- private static final ObjectWriter WRITER =
- new ObjectMapper().writerWithDefaultPrettyPrinter();
-
private final Text dtService;
// Allow fallback to default kms server port 9600 for certain tests that do
@@ -237,7 +234,7 @@ public class KMSClientProvider extends KeyProvider
implements CryptoExtension,
private static void writeJson(Object obj, OutputStream os)
throws IOException {
Writer writer = new OutputStreamWriter(os, StandardCharsets.UTF_8);
-WRITER.writeValue(writer, obj);
+JsonSerialization.writer().writeValue(writer, obj);
}
/**
http://git-wip-us.apache.org/repos/asf/hadoop/blob/89d44810/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
index 617773b..0ae2af3 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticator.java
@@ -17,8 +17,6 @@
*/
package org.apache.hadoop.security.token.delegation.web;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.databind.ObjectReader;
import org.apache.hadoop.classification.InterfaceAudience;
import
hadoop git commit: HADOOP-14501. Switch from aalto-xml to woodstox to handle odd XML features (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 6d7e847cf -> f901132a8
HADOOP-14501. Switch from aalto-xml to woodstox to handle odd XML features
(jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f901132a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f901132a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f901132a
Branch: refs/heads/branch-2
Commit: f901132a8affb7549c01bec8b3bf4137805fddeb
Parents: 6d7e847
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Jun 12 17:19:46 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Jun 12 17:19:46 2017 -0500
--
hadoop-common-project/hadoop-common/pom.xml | 4 +-
.../org/apache/hadoop/conf/Configuration.java | 6 +-
.../apache/hadoop/conf/TestConfiguration.java | 82
hadoop-project/pom.xml | 6 +-
4 files changed, 90 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f901132a/hadoop-common-project/hadoop-common/pom.xml
--
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index a2f5a14..b4d7120 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -302,8 +302,8 @@
compile
- com.fasterxml
- aalto-xml
+ com.fasterxml.woodstox
+ woodstox-core
compile
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f901132a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 9551e61..253c47a 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.conf;
-import com.fasterxml.aalto.stax.InputFactoryImpl;
+import com.ctc.wstx.stax.WstxInputFactory;
import com.google.common.annotations.VisibleForTesting;
import java.io.BufferedInputStream;
@@ -275,7 +275,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* Specify exact input factory to avoid time finding correct one.
* Factory is reusable across un-synchronized threads once initialized
*/
- private static final XMLInputFactory2 factory = new InputFactoryImpl();
+ private static final XMLInputFactory2 XML_INPUT_FACTORY = new
WstxInputFactory();
/**
* Class to keep the information about the keys which replace the deprecated
@@ -2570,7 +2570,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
if (is == null) {
return null;
}
-return factory.createXMLStreamReader(systemId, is);
+return XML_INPUT_FACTORY.createXMLStreamReader(systemId, is);
}
private void loadResources(Properties properties,
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f901132a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
index 135a4ca..fe121bc 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
@@ -30,6 +30,7 @@ import java.io.StringWriter;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.URI;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
@@ -99,6 +100,18 @@ public class TestConfiguration extends TestCase {
out.write("\n");
}
+ private void writeHeader() throws IOException{
+out.write("\n");
+ }
+
+ private void writeHeader(String encoding) throws IOException{
+out.write("\n");
+ }
+
+ private void writeConfiguration() throws IOException{
+out.write("\n");
+ }
+
private void endConfig() throws IOException{
out.write("\n");
out.close();
@@ -120,6 +133,18 @@ public class TestConfiguration extends TestCase
hadoop git commit: HADOOP-14501. Switch from aalto-xml to woodstox to handle odd XML features (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 86368cc76 -> a81916ea8
HADOOP-14501. Switch from aalto-xml to woodstox to handle odd XML features
(jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/a81916ea
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/a81916ea
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/a81916ea
Branch: refs/heads/trunk
Commit: a81916ea89d59c1642b3462e3d7c6c1acb1e7109
Parents: 86368cc
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Jun 12 17:07:53 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Jun 12 17:07:53 2017 -0500
--
hadoop-common-project/hadoop-common/pom.xml | 4 +-
.../org/apache/hadoop/conf/Configuration.java | 6 +-
.../apache/hadoop/conf/TestConfiguration.java | 82
hadoop-project/pom.xml | 6 +-
4 files changed, 90 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a81916ea/hadoop-common-project/hadoop-common/pom.xml
--
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 7ef0462..87b4dc9 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -314,8 +314,8 @@
compile
- com.fasterxml
- aalto-xml
+ com.fasterxml.woodstox
+ woodstox-core
compile
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a81916ea/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 1a6679b..d3dd822 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -18,7 +18,7 @@
package org.apache.hadoop.conf;
-import com.fasterxml.aalto.stax.InputFactoryImpl;
+import com.ctc.wstx.stax.WstxInputFactory;
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.google.common.annotations.VisibleForTesting;
@@ -284,7 +284,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* Specify exact input factory to avoid time finding correct one.
* Factory is reusable across un-synchronized threads once initialized
*/
- private static final XMLInputFactory2 factory = new InputFactoryImpl();
+ private static final XMLInputFactory2 XML_INPUT_FACTORY = new
WstxInputFactory();
/**
* Class to keep the information about the keys which replace the deprecated
@@ -2646,7 +2646,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
if (is == null) {
return null;
}
-return factory.createXMLStreamReader(systemId, is);
+return XML_INPUT_FACTORY.createXMLStreamReader(systemId, is);
}
private void loadResources(Properties properties,
http://git-wip-us.apache.org/repos/asf/hadoop/blob/a81916ea/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
index 0c66470..5ced541 100644
---
a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
+++
b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
@@ -30,6 +30,7 @@ import java.io.StringWriter;
import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.URI;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
@@ -99,6 +100,18 @@ public class TestConfiguration extends TestCase {
out.write("\n");
}
+ private void writeHeader() throws IOException{
+out.write("\n");
+ }
+
+ private void writeHeader(String encoding) throws IOException{
+out.write("\n");
+ }
+
+ private void writeConfiguration() throws IOException{
+out.write("\n");
+ }
+
private void endConfig() throws IOException{
out.write("\n");
out.close();
@@ -120,6 +133,18 @@ p
hadoop git commit: YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely Novák via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 1eecde335 -> f5f12b576
YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely
Novák via jeagles)
(cherry picked from commit cbfed0e82f57e96b8d5309e0613057963840554f)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f5f12b57
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f5f12b57
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f5f12b57
Branch: refs/heads/branch-2
Commit: f5f12b576efc7ab289fa88cca6e5760b5a7a12a9
Parents: 1eecde3
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed May 31 10:18:09 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed May 31 10:19:51 2017 -0500
--
.../server/resourcemanager/ResourceManager.java | 27 ++--
1 file changed, 13 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f5f12b57/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
index 3a93c6e..0573690 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
@@ -57,6 +57,7 @@ import org.apache.hadoop.yarn.event.AsyncDispatcher;
import org.apache.hadoop.yarn.event.Dispatcher;
import org.apache.hadoop.yarn.event.EventDispatcher;
import org.apache.hadoop.yarn.event.EventHandler;
+import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import
org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter;
import
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventType;
@@ -220,13 +221,7 @@ public class ResourceManager extends CompositeService
implements Recoverable {
rmContext.setConfigurationProvider(configurationProvider);
// load core-site.xml
-InputStream coreSiteXMLInputStream =
-this.configurationProvider.getConfigurationInputStream(this.conf,
-YarnConfiguration.CORE_SITE_CONFIGURATION_FILE);
-if (coreSiteXMLInputStream != null) {
- this.conf.addResource(coreSiteXMLInputStream,
- YarnConfiguration.CORE_SITE_CONFIGURATION_FILE);
-}
+loadConfigurationXml(YarnConfiguration.CORE_SITE_CONFIGURATION_FILE);
// Do refreshUserToGroupsMappings with loaded core-site.xml
Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(this.conf)
@@ -239,13 +234,7 @@ public class ResourceManager extends CompositeService
implements Recoverable {
ProxyUsers.refreshSuperUserGroupsConfiguration(this.conf);
// load yarn-site.xml
-InputStream yarnSiteXMLInputStream =
-this.configurationProvider.getConfigurationInputStream(this.conf,
-YarnConfiguration.YARN_SITE_CONFIGURATION_FILE);
-if (yarnSiteXMLInputStream != null) {
- this.conf.addResource(yarnSiteXMLInputStream,
- YarnConfiguration.YARN_SITE_CONFIGURATION_FILE);
-}
+loadConfigurationXml(YarnConfiguration.YARN_SITE_CONFIGURATION_FILE);
validateConfigs(this.conf);
@@ -311,6 +300,16 @@ public class ResourceManager extends CompositeService
implements Recoverable {
super.serviceInit(this.conf);
}
+ private void loadConfigurationXml(String configurationFile)
+ throws YarnException, IOException {
+InputStream configurationInputStream =
+this.configurationProvider.getConfigurationInputStream(this.conf,
+configurationFile);
+if (configurationInputStream != null) {
+ this.conf.addResource(configurationInputStream, configurationFile);
+}
+ }
+
protected EmbeddedElector createEmbeddedElector() throws IOException {
EmbeddedElector elector;
curatorEnabled =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely Novák via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 13de636b4 -> cbfed0e82
YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely
Novák via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/cbfed0e8
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/cbfed0e8
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/cbfed0e8
Branch: refs/heads/trunk
Commit: cbfed0e82f57e96b8d5309e0613057963840554f
Parents: 13de636
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed May 31 10:18:09 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed May 31 10:18:42 2017 -0500
--
.../server/resourcemanager/ResourceManager.java | 27 ++--
1 file changed, 13 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/cbfed0e8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
index 8f2c121..f727f55 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
@@ -59,6 +59,7 @@ import org.apache.hadoop.yarn.event.AsyncDispatcher;
import org.apache.hadoop.yarn.event.Dispatcher;
import org.apache.hadoop.yarn.event.EventDispatcher;
import org.apache.hadoop.yarn.event.EventHandler;
+import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import
org.apache.hadoop.yarn.server.resourcemanager.ahs.RMApplicationHistoryWriter;
import
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncherEventType;
@@ -238,13 +239,7 @@ public class ResourceManager extends CompositeService
implements Recoverable {
rmContext.setConfigurationProvider(configurationProvider);
// load core-site.xml
-InputStream coreSiteXMLInputStream =
-this.configurationProvider.getConfigurationInputStream(this.conf,
-YarnConfiguration.CORE_SITE_CONFIGURATION_FILE);
-if (coreSiteXMLInputStream != null) {
- this.conf.addResource(coreSiteXMLInputStream,
- YarnConfiguration.CORE_SITE_CONFIGURATION_FILE);
-}
+loadConfigurationXml(YarnConfiguration.CORE_SITE_CONFIGURATION_FILE);
// Do refreshUserToGroupsMappings with loaded core-site.xml
Groups.getUserToGroupsMappingServiceWithLoadedConfiguration(this.conf)
@@ -257,13 +252,7 @@ public class ResourceManager extends CompositeService
implements Recoverable {
ProxyUsers.refreshSuperUserGroupsConfiguration(this.conf);
// load yarn-site.xml
-InputStream yarnSiteXMLInputStream =
-this.configurationProvider.getConfigurationInputStream(this.conf,
-YarnConfiguration.YARN_SITE_CONFIGURATION_FILE);
-if (yarnSiteXMLInputStream != null) {
- this.conf.addResource(yarnSiteXMLInputStream,
- YarnConfiguration.YARN_SITE_CONFIGURATION_FILE);
-}
+loadConfigurationXml(YarnConfiguration.YARN_SITE_CONFIGURATION_FILE);
validateConfigs(this.conf);
@@ -339,6 +328,16 @@ public class ResourceManager extends CompositeService
implements Recoverable {
super.serviceInit(this.conf);
}
+ private void loadConfigurationXml(String configurationFile)
+ throws YarnException, IOException {
+InputStream configurationInputStream =
+this.configurationProvider.getConfigurationInputStream(this.conf,
+configurationFile);
+if (configurationInputStream != null) {
+ this.conf.addResource(configurationInputStream, configurationFile);
+}
+ }
+
protected EmbeddedElector createEmbeddedElector() throws IOException {
EmbeddedElector elector;
curatorEnabled =
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.8.1 486d1a635 -> 717cd78f5
YARN-6598. History server getApplicationReport NPE when fetching report for
pre-2.8 job (Jason Lowe via jeagles)
(cherry picked from commit c992cf638e9b391dd8ad5dbb87df59c187a98f18)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/717cd78f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/717cd78f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/717cd78f
Branch: refs/heads/branch-2.8.1
Commit: 717cd78f5734734d4d1237c5e1581eb5ca594638
Parents: 486d1a6
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon May 15 12:10:12 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon May 15 12:11:53 2017 -0500
--
...pplicationHistoryManagerOnTimelineStore.java | 35
...pplicationHistoryManagerOnTimelineStore.java | 31 +
2 files changed, 47 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/717cd78f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index 3a11c92..7434178 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -330,20 +330,19 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
}
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_CPU_METRICS))
{
-long vcoreSeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_CPU_METRICS).toString());
-long memorySeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_MEM_METRICS).toString());
-long preemptedMemorySeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_MEM_PREEMPT_METRICS).toString());
-long preemptedVcoreSeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_CPU_PREEMPT_METRICS).toString());
-appResources = ApplicationResourceUsageReport
-.newInstance(0, 0, null, null, null, memorySeconds, vcoreSeconds,
0,
-0, preemptedMemorySeconds, preemptedVcoreSeconds);
+long vcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_METRICS);
+long memorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_METRICS);
+long preemptedMemorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_PREEMPT_METRICS);
+long preemptedVcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_PREEMPT_METRICS);
+appResources = ApplicationResourceUsageReport.newInstance(0, 0, null,
+null, null, memorySeconds, vcoreSeconds, 0, 0,
+preemptedMemorySeconds, preemptedVcoreSeconds);
}
+
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_TAGS_INFO)) {
appTags = new HashSet();
Object obj = entityInfo.get(ApplicationMetricsConstants.APP_TAGS_INFO);
@@ -445,6 +444,16 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
amNodeLabelExpression), appViewACLs);
}
+ private static long parseLong(Map<String, Object> entityInfo,
+ String infoKey) {
+long result = 0;
+Object infoValue = entityInfo.get(infoKey);
+if (infoValue != null) {
+ result = Long.parseLong(infoValue.toString());
+}
+return result;
+ }
+
private static boolean isFinalState(YarnApplicationState state) {
return state == YarnApplicationState.FINISHED
|| state == YarnApplicationState.FAILED
http://git-wip-us.apache.org/repos/asf/hadoop/blob/717cd78f/hadoop-yarn-proje
hadoop git commit: YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.8 f4a75 -> c992cf638
YARN-6598. History server getApplicationReport NPE when fetching report for
pre-2.8 job (Jason Lowe via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c992cf63
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c992cf63
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c992cf63
Branch: refs/heads/branch-2.8
Commit: c992cf638e9b391dd8ad5dbb87df59c187a98f18
Parents: f4a
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon May 15 12:10:12 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon May 15 12:10:12 2017 -0500
--
...pplicationHistoryManagerOnTimelineStore.java | 35
...pplicationHistoryManagerOnTimelineStore.java | 31 +
2 files changed, 47 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c992cf63/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index 3a11c92..7434178 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -330,20 +330,19 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
}
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_CPU_METRICS))
{
-long vcoreSeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_CPU_METRICS).toString());
-long memorySeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_MEM_METRICS).toString());
-long preemptedMemorySeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_MEM_PREEMPT_METRICS).toString());
-long preemptedVcoreSeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_CPU_PREEMPT_METRICS).toString());
-appResources = ApplicationResourceUsageReport
-.newInstance(0, 0, null, null, null, memorySeconds, vcoreSeconds,
0,
-0, preemptedMemorySeconds, preemptedVcoreSeconds);
+long vcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_METRICS);
+long memorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_METRICS);
+long preemptedMemorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_PREEMPT_METRICS);
+long preemptedVcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_PREEMPT_METRICS);
+appResources = ApplicationResourceUsageReport.newInstance(0, 0, null,
+null, null, memorySeconds, vcoreSeconds, 0, 0,
+preemptedMemorySeconds, preemptedVcoreSeconds);
}
+
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_TAGS_INFO)) {
appTags = new HashSet();
Object obj = entityInfo.get(ApplicationMetricsConstants.APP_TAGS_INFO);
@@ -445,6 +444,16 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
amNodeLabelExpression), appViewACLs);
}
+ private static long parseLong(Map<String, Object> entityInfo,
+ String infoKey) {
+long result = 0;
+Object infoValue = entityInfo.get(infoKey);
+if (infoValue != null) {
+ result = Long.parseLong(infoValue.toString());
+}
+return result;
+ }
+
private static boolean isFinalState(YarnApplicationState state) {
return state == YarnApplicationState.FINISHED
|| state == YarnApplicationState.FAILED
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c992cf63/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistory
hadoop git commit: YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 18b89e902 -> d370edc6a
YARN-6598. History server getApplicationReport NPE when fetching report for
pre-2.8 job (Jason Lowe via jeagles)
(cherry picked from commit c48f2976a3de60b95c4a5ada4f0131c4cdde177a)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d370edc6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d370edc6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d370edc6
Branch: refs/heads/branch-2
Commit: d370edc6aa72ec34e9bac8afef09c775c0e70238
Parents: 18b89e9
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon May 15 10:32:01 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon May 15 10:32:47 2017 -0500
--
...pplicationHistoryManagerOnTimelineStore.java | 35
...pplicationHistoryManagerOnTimelineStore.java | 32 ++
2 files changed, 48 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d370edc6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index d47c43a..4bc821c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -330,20 +330,19 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
}
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_CPU_METRICS))
{
-long vcoreSeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_CPU_METRICS).toString());
-long memorySeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_MEM_METRICS).toString());
-long preemptedMemorySeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_MEM_PREEMPT_METRICS).toString());
-long preemptedVcoreSeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_CPU_PREEMPT_METRICS).toString());
-appResources = ApplicationResourceUsageReport
-.newInstance(0, 0, null, null, null, memorySeconds, vcoreSeconds,
0,
-0, preemptedMemorySeconds, preemptedVcoreSeconds);
+long vcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_METRICS);
+long memorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_METRICS);
+long preemptedMemorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_PREEMPT_METRICS);
+long preemptedVcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_PREEMPT_METRICS);
+appResources = ApplicationResourceUsageReport.newInstance(0, 0, null,
+null, null, memorySeconds, vcoreSeconds, 0, 0,
+preemptedMemorySeconds, preemptedVcoreSeconds);
}
+
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_TAGS_INFO)) {
appTags = new HashSet();
Object obj = entityInfo.get(ApplicationMetricsConstants.APP_TAGS_INFO);
@@ -445,6 +444,16 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
amNodeLabelExpression), appViewACLs);
}
+ private static long parseLong(Map<String, Object> entityInfo,
+ String infoKey) {
+long result = 0;
+Object infoValue = entityInfo.get(infoKey);
+if (infoValue != null) {
+ result = Long.parseLong(infoValue.toString());
+}
+return result;
+ }
+
private static boolean isFinalState(YarnApplicationState state) {
return state == YarnApplicationState.FINISHED
|| state == YarnApplicationState.FAILED
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d370edc6/hadoop-yarn-project/ha
hadoop git commit: YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 6600abbb5 -> c48f2976a
YARN-6598. History server getApplicationReport NPE when fetching report for
pre-2.8 job (Jason Lowe via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c48f2976
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c48f2976
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c48f2976
Branch: refs/heads/trunk
Commit: c48f2976a3de60b95c4a5ada4f0131c4cdde177a
Parents: 6600abb
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon May 15 10:32:01 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon May 15 10:32:01 2017 -0500
--
...pplicationHistoryManagerOnTimelineStore.java | 35
...pplicationHistoryManagerOnTimelineStore.java | 32 ++
2 files changed, 48 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c48f2976/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index 54893ef..d18f3dc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -330,20 +330,19 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
}
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_CPU_METRICS))
{
-long vcoreSeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_CPU_METRICS).toString());
-long memorySeconds=Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants.APP_MEM_METRICS).toString());
-long preemptedMemorySeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_MEM_PREEMPT_METRICS).toString());
-long preemptedVcoreSeconds = Long.parseLong(entityInfo.get(
-ApplicationMetricsConstants
-.APP_CPU_PREEMPT_METRICS).toString());
-appResources = ApplicationResourceUsageReport
-.newInstance(0, 0, null, null, null, memorySeconds, vcoreSeconds,
0,
-0, preemptedMemorySeconds, preemptedVcoreSeconds);
+long vcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_METRICS);
+long memorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_METRICS);
+long preemptedMemorySeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_MEM_PREEMPT_METRICS);
+long preemptedVcoreSeconds = parseLong(entityInfo,
+ApplicationMetricsConstants.APP_CPU_PREEMPT_METRICS);
+appResources = ApplicationResourceUsageReport.newInstance(0, 0, null,
+null, null, memorySeconds, vcoreSeconds, 0, 0,
+preemptedMemorySeconds, preemptedVcoreSeconds);
}
+
if (entityInfo.containsKey(ApplicationMetricsConstants.APP_TAGS_INFO)) {
appTags = new HashSet();
Object obj = entityInfo.get(ApplicationMetricsConstants.APP_TAGS_INFO);
@@ -445,6 +444,16 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
amNodeLabelExpression), appViewACLs);
}
+ private static long parseLong(Map<String, Object> entityInfo,
+ String infoKey) {
+long result = 0;
+Object infoValue = entityInfo.get(infoKey);
+if (infoValue != null) {
+ result = Long.parseLong(infoValue.toString());
+}
+return result;
+ }
+
private static boolean isFinalState(YarnApplicationState state) {
return state == YarnApplicationState.FINISHED
|| state == YarnApplicationState.FAILED
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c48f2976/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src
hadoop git commit: HADOOP-14216. Addendum to Improve Configuration XML Parsing Performance (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 a52b30f7e -> 60e65a3e5
HADOOP-14216. Addendum to Improve Configuration XML Parsing Performance
(jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/60e65a3e
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/60e65a3e
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/60e65a3e
Branch: refs/heads/branch-2
Commit: 60e65a3e551ed96c39d21b2a4b5cea17289e40d9
Parents: a52b30f
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Thu Mar 30 14:15:37 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Thu Mar 30 14:15:37 2017 -0500
--
.../src/main/java/org/apache/hadoop/conf/Configuration.java | 9 -
1 file changed, 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/60e65a3e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index fee84c2..df28b96 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -2678,9 +2678,6 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
token.setLength(0);
break;
case "include":
-if (!"xi".equals(reader.getPrefix())) {
- break;
-}
// Determine href for xi:include
String confInclude = null;
attrCount = reader.getAttributeCount();
@@ -2717,9 +2714,6 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
}
break;
case "fallback":
-if (!"xi".equals(reader.getPrefix())) {
- break;
-}
fallbackEntered = true;
break;
case "configuration":
@@ -2755,9 +2749,6 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
confSource.add(StringInterner.weakIntern(token.toString()));
break;
case "include":
-if (!"xi".equals(reader.getPrefix())) {
- break;
-}
if (fallbackAllowed && !fallbackEntered) {
throw new IOException("Fetch fail on include with no "
+ "fallback while loading '" + name + "'");
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-14216. Addendum to Improve Configuration XML Parsing Performance (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 8c591b8d1 -> 1309c585f
HADOOP-14216. Addendum to Improve Configuration XML Parsing Performance
(jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/1309c585
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/1309c585
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/1309c585
Branch: refs/heads/trunk
Commit: 1309c585fb9f632f7c649464ecbe358c5130b142
Parents: 8c591b8
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Thu Mar 30 14:14:43 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Thu Mar 30 14:14:43 2017 -0500
--
.../src/main/java/org/apache/hadoop/conf/Configuration.java | 9 -
1 file changed, 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/1309c585/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 36845e3..2ac52cb 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -2756,9 +2756,6 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
token.setLength(0);
break;
case "include":
-if (!"xi".equals(reader.getPrefix())) {
- break;
-}
// Determine href for xi:include
String confInclude = null;
attrCount = reader.getAttributeCount();
@@ -2795,9 +2792,6 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
}
break;
case "fallback":
-if (!"xi".equals(reader.getPrefix())) {
- break;
-}
fallbackEntered = true;
break;
case "configuration":
@@ -2833,9 +2827,6 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
confSource.add(StringInterner.weakIntern(token.toString()));
break;
case "include":
-if (!"xi".equals(reader.getPrefix())) {
- break;
-}
if (fallbackAllowed && !fallbackEntered) {
throw new IOException("Fetch fail on include with no "
+ "fallback while loading '" + name + "'");
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: HADOOP-14216. Improve Configuration XML Parsing Performance (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 82b4a9c3d -> 719ae5f2e
HADOOP-14216. Improve Configuration XML Parsing Performance (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/719ae5f2
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/719ae5f2
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/719ae5f2
Branch: refs/heads/branch-2
Commit: 719ae5f2e8a73528a7d3e2955a8163bbfd4a28d8
Parents: 82b4a9c
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed Mar 29 10:14:49 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed Mar 29 10:14:49 2017 -0500
--
hadoop-common-project/hadoop-common/pom.xml | 10 +
.../org/apache/hadoop/conf/Configuration.java | 301 ---
.../apache/hadoop/conf/TestConfiguration.java | 62 +++-
hadoop-project/pom.xml | 10 +
4 files changed, 266 insertions(+), 117 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/719ae5f2/hadoop-common-project/hadoop-common/pom.xml
--
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index ee7acb3..8b39a7f 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -296,6 +296,16 @@
bcprov-jdk16
test
+
+ org.codehaus.woodstox
+ stax2-api
+ compile
+
+
+ com.fasterxml
+ aalto-xml
+ compile
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/719ae5f2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 9301f51..fee84c2 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.conf;
+import com.fasterxml.aalto.stax.InputFactoryImpl;
import com.google.common.annotations.VisibleForTesting;
import java.io.BufferedInputStream;
@@ -63,9 +64,11 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
-import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.stream.XMLStreamConstants;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamReader;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
@@ -93,13 +96,10 @@ import org.apache.hadoop.util.StringInterner;
import org.apache.hadoop.util.StringUtils;
import org.codehaus.jackson.JsonFactory;
import org.codehaus.jackson.JsonGenerator;
-import org.w3c.dom.DOMException;
+import org.codehaus.stax2.XMLInputFactory2;
+import org.codehaus.stax2.XMLStreamReader2;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
-import org.w3c.dom.Node;
-import org.w3c.dom.NodeList;
-import org.w3c.dom.Text;
-import org.xml.sax.SAXException;
import com.google.common.base.Preconditions;
@@ -270,7 +270,13 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* the key most recently
*/
private Map<String, String[]> updatingResource;
-
+
+ /**
+ * Specify exact input factory to avoid time finding correct one.
+ * Factory is reusable across un-synchronized threads once initialized
+ */
+ private static final XMLInputFactory2 factory = new InputFactoryImpl();
+
/**
* Class to keep the information about the keys which replace the deprecated
* ones.
@@ -681,7 +687,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
addDefaultResource("hadoop-site.xml");
}
}
-
+
private Properties properties;
private Properties overlay;
private ClassLoader classLoader;
@@ -2534,8 +2540,8 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
return configMap;
}
- private Document parse(DocumentBuilder builder, URL url)
- throws IOException, SAXException {
+ private XMLStreamReader parse(URL url)
+ throws IOException, XMLStreamException {
if (!quietmode) {
if (LOG.isDebugEnabled()) {
LOG.debug("
hadoop git commit: HADOOP-14216. Improve Configuration XML Parsing Performance (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 84d787b9d -> 523f467d9
HADOOP-14216. Improve Configuration XML Parsing Performance (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/523f467d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/523f467d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/523f467d
Branch: refs/heads/trunk
Commit: 523f467d939d80e2bc162e1f47be497109783061
Parents: 84d787b
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed Mar 29 10:12:02 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed Mar 29 10:12:02 2017 -0500
--
hadoop-common-project/hadoop-common/pom.xml | 10 +
.../org/apache/hadoop/conf/Configuration.java | 317 +++
.../apache/hadoop/conf/TestConfiguration.java | 62 +++-
hadoop-project/pom.xml | 10 +
4 files changed, 266 insertions(+), 133 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/523f467d/hadoop-common-project/hadoop-common/pom.xml
--
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 961c902..ee82df0 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -313,6 +313,16 @@
com.fasterxml.jackson.core
jackson-databind
+
+ org.codehaus.woodstox
+ stax2-api
+ compile
+
+
+ com.fasterxml
+ aalto-xml
+ compile
+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/523f467d/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index da4f6ac..36845e3 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -18,6 +18,7 @@
package org.apache.hadoop.conf;
+import com.fasterxml.aalto.stax.InputFactoryImpl;
import com.fasterxml.jackson.core.JsonFactory;
import com.fasterxml.jackson.core.JsonGenerator;
import com.google.common.annotations.VisibleForTesting;
@@ -65,9 +66,11 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
-import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
+import javax.xml.stream.XMLStreamConstants;
+import javax.xml.stream.XMLStreamException;
+import javax.xml.stream.XMLStreamReader;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
@@ -93,14 +96,10 @@ import
org.apache.hadoop.security.alias.CredentialProviderFactory;
import org.apache.hadoop.util.ReflectionUtils;
import org.apache.hadoop.util.StringInterner;
import org.apache.hadoop.util.StringUtils;
-import org.w3c.dom.Attr;
-import org.w3c.dom.DOMException;
+import org.codehaus.stax2.XMLInputFactory2;
+import org.codehaus.stax2.XMLStreamReader2;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
-import org.w3c.dom.Node;
-import org.w3c.dom.NodeList;
-import org.w3c.dom.Text;
-import org.xml.sax.SAXException;
import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
@@ -280,7 +279,13 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* the key most recently
*/
private Map<String, String[]> updatingResource;
-
+
+ /**
+ * Specify exact input factory to avoid time finding correct one.
+ * Factory is reusable across un-synchronized threads once initialized
+ */
+ private static final XMLInputFactory2 factory = new InputFactoryImpl();
+
/**
* Class to keep the information about the keys which replace the deprecated
* ones.
@@ -689,7 +694,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
addDefaultResource("hadoop-site.xml");
}
}
-
+
private Properties properties;
private Properties overlay;
private ClassLoader classLoader;
@@ -2613,8 +2618,8 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
return configMap;
}
- private Document parse(DocumentBuilder builder, URL url)
- throws IOException, SAXException {
hadoop git commit: Configuration#get return value optimization (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.8 4e40e9d4d -> ec5a65a0c
Configuration#get return value optimization (jeagles)
(cherry picked from commit e41aac0cd41c6a4ee740afd09126bc48072880ab)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ec5a65a0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ec5a65a0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ec5a65a0
Branch: refs/heads/branch-2.8
Commit: ec5a65a0ca0aa51812b8fa2f613c5c031b446e7a
Parents: 4e40e9d
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Mar 27 12:52:59 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Mar 27 12:53:42 2017 -0500
--
.../org/apache/hadoop/conf/Configuration.java | 54 +++-
1 file changed, 29 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/ec5a65a0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 7d10471..0573b13 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -610,37 +610,44 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* deprecated key, the value of the deprecated key is set as the value for
* the provided property name.
*
+ * @param deprecations deprecation context
* @param name the property name
* @return the first property in the list of properties mapping
* the name or the name itself.
*/
private String[] handleDeprecation(DeprecationContext deprecations,
- String name) {
+ String name) {
if (null != name) {
name = name.trim();
}
-ArrayList names = new ArrayList();
- if (isDeprecated(name)) {
- DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
- warnOnceIfDeprecated(deprecations, name);
- for (String newKey : keyInfo.newKeys) {
-if(newKey != null) {
- names.add(newKey);
-}
+// Initialize the return value with requested name
+String[] names = new String[]{name};
+// Deprecated keys are logged once and an updated names are returned
+DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
+if (keyInfo != null) {
+ if (!keyInfo.getAndSetAccessed()) {
+logDeprecation(keyInfo.getWarningMessage(name));
}
+ // Override return value for deprecated keys
+ names = keyInfo.newKeys;
}
-if(names.size() == 0) {
- names.add(name);
-}
-for(String n : names) {
- String deprecatedKey =
deprecations.getReverseDeprecatedKeyMap().get(n);
- if (deprecatedKey != null && !getOverlay().containsKey(n) &&
- getOverlay().containsKey(deprecatedKey)) {
- getProps().setProperty(n, getOverlay().getProperty(deprecatedKey));
- getOverlay().setProperty(n,
getOverlay().getProperty(deprecatedKey));
- }
+// If there are no overlay values we can return early
+Properties overlay = getOverlay();
+if (overlay.isEmpty()) {
+ return names;
+}
+// Update properties and overlays with reverse lookup values
+for (String n : names) {
+ String deprecatedKey = deprecations.getReverseDeprecatedKeyMap().get(n);
+ if (deprecatedKey != null && !overlay.containsKey(n)) {
+String deprecatedValue = overlay.getProperty(deprecatedKey);
+if (deprecatedValue != null) {
+ getProps().setProperty(n, deprecatedValue);
+ overlay.setProperty(n, deprecatedValue);
+}
+ }
}
-return names.toArray(new String[names.size()]);
+return names;
}
private void handleDeprecation() {
@@ -1175,11 +1182,8 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
}
}
- private void warnOnceIfDeprecated(DeprecationContext deprecations, String
name) {
-DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
-if (keyInfo != null && !keyInfo.getAndSetAccessed()) {
- LOG_DEPRECATION.info(keyInfo.getWarningMessage(name));
-}
+ private void logDeprecation(String message) {
+LOG_DEPRECATION.info(message);
}
/**
-
To unsubscri
hadoop git commit: Configuration#get return value optimization (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 675bc97f1 -> e41aac0cd
Configuration#get return value optimization (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/e41aac0c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/e41aac0c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/e41aac0c
Branch: refs/heads/branch-2
Commit: e41aac0cd41c6a4ee740afd09126bc48072880ab
Parents: 675bc97
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Mar 27 12:52:59 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Mar 27 12:52:59 2017 -0500
--
.../org/apache/hadoop/conf/Configuration.java | 54 +++-
1 file changed, 29 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/e41aac0c/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index a2d0f44..87b608b 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -611,37 +611,44 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* deprecated key, the value of the deprecated key is set as the value for
* the provided property name.
*
+ * @param deprecations deprecation context
* @param name the property name
* @return the first property in the list of properties mapping
* the name or the name itself.
*/
private String[] handleDeprecation(DeprecationContext deprecations,
- String name) {
+ String name) {
if (null != name) {
name = name.trim();
}
-ArrayList names = new ArrayList();
- if (isDeprecated(name)) {
- DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
- warnOnceIfDeprecated(deprecations, name);
- for (String newKey : keyInfo.newKeys) {
-if(newKey != null) {
- names.add(newKey);
-}
+// Initialize the return value with requested name
+String[] names = new String[]{name};
+// Deprecated keys are logged once and an updated names are returned
+DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
+if (keyInfo != null) {
+ if (!keyInfo.getAndSetAccessed()) {
+logDeprecation(keyInfo.getWarningMessage(name));
}
+ // Override return value for deprecated keys
+ names = keyInfo.newKeys;
}
-if(names.size() == 0) {
- names.add(name);
-}
-for(String n : names) {
- String deprecatedKey =
deprecations.getReverseDeprecatedKeyMap().get(n);
- if (deprecatedKey != null && !getOverlay().containsKey(n) &&
- getOverlay().containsKey(deprecatedKey)) {
- getProps().setProperty(n, getOverlay().getProperty(deprecatedKey));
- getOverlay().setProperty(n,
getOverlay().getProperty(deprecatedKey));
- }
+// If there are no overlay values we can return early
+Properties overlay = getOverlay();
+if (overlay.isEmpty()) {
+ return names;
+}
+// Update properties and overlays with reverse lookup values
+for (String n : names) {
+ String deprecatedKey = deprecations.getReverseDeprecatedKeyMap().get(n);
+ if (deprecatedKey != null && !overlay.containsKey(n)) {
+String deprecatedValue = overlay.getProperty(deprecatedKey);
+if (deprecatedValue != null) {
+ getProps().setProperty(n, deprecatedValue);
+ overlay.setProperty(n, deprecatedValue);
+}
+ }
}
-return names.toArray(new String[names.size()]);
+return names;
}
private void handleDeprecation() {
@@ -1179,11 +1186,8 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
}
}
- private void warnOnceIfDeprecated(DeprecationContext deprecations, String
name) {
-DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
-if (keyInfo != null && !keyInfo.getAndSetAccessed()) {
- LOG_DEPRECATION.info(keyInfo.getWarningMessage(name));
-}
+ private void logDeprecation(String message) {
+LOG_DEPRECATION.info(message);
}
/**
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: Configuration#get return value optimization (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 858d597be -> db2adf356
Configuration#get return value optimization (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/db2adf35
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/db2adf35
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/db2adf35
Branch: refs/heads/trunk
Commit: db2adf356a937e4c539b2cf5dccfaf90271cf43e
Parents: 858d597
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Mar 27 12:48:44 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Mar 27 12:48:44 2017 -0500
--
.../org/apache/hadoop/conf/Configuration.java | 52 ++--
1 file changed, 27 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/db2adf35/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 60b0398..ada8b02 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -619,42 +619,44 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
* deprecated key, the value of the deprecated key is set as the value for
* the provided property name.
*
+ * @param deprecations deprecation context
* @param name the property name
* @return the first property in the list of properties mapping
* the name or the name itself.
*/
private String[] handleDeprecation(DeprecationContext deprecations,
- String name) {
+ String name) {
if (null != name) {
name = name.trim();
}
-ArrayList names = new ArrayList();
- if (isDeprecated(name)) {
- DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
- if (keyInfo != null) {
-if (!keyInfo.getAndSetAccessed()) {
- logDeprecation(keyInfo.getWarningMessage(name));
-}
-
-for (String newKey : keyInfo.newKeys) {
- if (newKey != null) {
-names.add(newKey);
- }
+// Initialize the return value with requested name
+String[] names = new String[]{name};
+// Deprecated keys are logged once and an updated names are returned
+DeprecatedKeyInfo keyInfo = deprecations.getDeprecatedKeyMap().get(name);
+if (keyInfo != null) {
+ if (!keyInfo.getAndSetAccessed()) {
+logDeprecation(keyInfo.getWarningMessage(name));
+ }
+ // Override return value for deprecated keys
+ names = keyInfo.newKeys;
+}
+// If there are no overlay values we can return early
+Properties overlayProperties = getOverlay();
+if (overlayProperties.isEmpty()) {
+ return names;
+}
+// Update properties and overlays with reverse lookup values
+for (String n : names) {
+ String deprecatedKey = deprecations.getReverseDeprecatedKeyMap().get(n);
+ if (deprecatedKey != null && !overlayProperties.containsKey(n)) {
+String deprecatedValue = overlayProperties.getProperty(deprecatedKey);
+if (deprecatedValue != null) {
+ getProps().setProperty(n, deprecatedValue);
+ overlayProperties.setProperty(n, deprecatedValue);
}
}
}
-if(names.size() == 0) {
- names.add(name);
-}
-for(String n : names) {
- String deprecatedKey =
deprecations.getReverseDeprecatedKeyMap().get(n);
- if (deprecatedKey != null && !getOverlay().containsKey(n) &&
- getOverlay().containsKey(deprecatedKey)) {
- getProps().setProperty(n, getOverlay().getProperty(deprecatedKey));
- getOverlay().setProperty(n,
getOverlay().getProperty(deprecatedKey));
- }
-}
-return names.toArray(new String[names.size()]);
+return names;
}
private void handleDeprecation() {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: Delay construction of PreCondition.check failure message in Configuration#set (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.8 19d479168 -> 4e40e9d4d
Delay construction of PreCondition.check failure message in Configuration#set
(jeagles)
(cherry picked from commit 858d597be00ce006c1d1d4cd476729e4f1de8d25)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4e40e9d4
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4e40e9d4
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4e40e9d4
Branch: refs/heads/branch-2.8
Commit: 4e40e9d4db6cf6c8b53a782113df9fc460335817
Parents: 19d4791
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Mar 27 11:01:48 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Mar 27 11:06:41 2017 -0500
--
.../src/main/java/org/apache/hadoop/conf/Configuration.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4e40e9d4/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index f016119..7d10471 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -1141,7 +1141,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
"Property name must not be null");
Preconditions.checkArgument(
value != null,
-"The value of property " + name + " must not be null");
+"The value of property %s must not be null", name);
name = name.trim();
DeprecationContext deprecations = deprecationContext.get();
if (deprecations.getDeprecatedKeyMap().isEmpty()) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: Delay construction of PreCondition.check failure message in Configuration#set (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 0e8beb214 -> 675bc97f1
Delay construction of PreCondition.check failure message in Configuration#set
(jeagles)
(cherry picked from commit 858d597be00ce006c1d1d4cd476729e4f1de8d25)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/675bc97f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/675bc97f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/675bc97f
Branch: refs/heads/branch-2
Commit: 675bc97f18b303a9853a16768545bb519c73cbaf
Parents: 0e8beb2
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Mar 27 11:01:48 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Mar 27 11:02:41 2017 -0500
--
.../src/main/java/org/apache/hadoop/conf/Configuration.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/675bc97f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index 2528da9..a2d0f44 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -1145,7 +1145,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
"Property name must not be null");
Preconditions.checkArgument(
value != null,
-"The value of property " + name + " must not be null");
+"The value of property %s must not be null", name);
name = name.trim();
DeprecationContext deprecations = deprecationContext.get();
if (deprecations.getDeprecatedKeyMap().isEmpty()) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
hadoop git commit: Delay construction of PreCondition.check failure message in Configuration#set (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 46d37a65c -> 858d597be
Delay construction of PreCondition.check failure message in Configuration#set
(jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/858d597b
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/858d597b
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/858d597b
Branch: refs/heads/trunk
Commit: 858d597be00ce006c1d1d4cd476729e4f1de8d25
Parents: 46d37a6
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Mar 27 11:01:48 2017 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Mar 27 11:01:48 2017 -0500
--
.../src/main/java/org/apache/hadoop/conf/Configuration.java| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/858d597b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
index a9c8d9c..60b0398 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
@@ -1208,7 +1208,7 @@ public class Configuration implements
Iterable<Map.Entry<String,String>>,
"Property name must not be null");
Preconditions.checkArgument(
value != null,
-"The value of property " + name + " must not be null");
+"The value of property %s must not be null", name);
name = name.trim();
DeprecationContext deprecations = deprecationContext.get();
if (deprecations.getDeprecatedKeyMap().isEmpty()) {
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[2/2] hadoop git commit: YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App page (Eric Payne via jeagles)
YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App
page (Eric Payne via jeagles)
(cherry picked from commit 4ff973f96ae7f77cda3b52b38427e2991819ad31)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7c604c3d
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7c604c3d
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7c604c3d
Branch: refs/heads/branch-2
Commit: 7c604c3d065ab15927ef7e14be6acf7772f6077f
Parents: 7decba8
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Dec 7 15:04:48 2015 -0600
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Dec 7 15:05:15 2015 -0600
--
hadoop-yarn-project/CHANGES.txt | 3 +++
.../ApplicationHistoryManagerOnTimelineStore.java | 7 +++
.../resourcemanager/metrics/AppAttemptFinishedEvent.java | 10 +-
.../resourcemanager/metrics/SystemMetricsPublisher.java | 6 +-
4 files changed, 24 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7c604c3d/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 381ab7f..f956cbd 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -1097,6 +1097,9 @@ Release 2.7.3 - UNRELEASED
(with application having id > ) （Mohammad Shahid Khan & Varun Saxena
via jianhe)
+YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on
App page
+ (Eric Payne via jeagles)
+
Release 2.7.2 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7c604c3d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index 9b4e78e..6e6b9fc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -488,6 +488,13 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
AppAttemptMetricsConstants.STATE_EVENT_INFO)
.toString());
}
+ if (eventInfo
+
.containsKey(AppAttemptMetricsConstants.MASTER_CONTAINER_EVENT_INFO)) {
+amContainerId =
+ConverterUtils.toContainerId(eventInfo.get(
+AppAttemptMetricsConstants.MASTER_CONTAINER_EVENT_INFO)
+.toString());
+ }
}
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7c604c3d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
index 71d9363..fc1d10f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.resourcemanager.metrics;
import org.apache.hadoop.yarn.api.records.Applicatio
hadoop git commit: YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App page (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.7 670617f32 -> 22be47d74
YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App
page (Eric Payne via jeagles)
(cherry picked from commit 4ff973f96ae7f77cda3b52b38427e2991819ad31)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/22be47d7
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/22be47d7
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/22be47d7
Branch: refs/heads/branch-2.7
Commit: 22be47d74768f292392a338f49d14ddda1f918fe
Parents: 670617f
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Mon Dec 7 15:04:48 2015 -0600
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Mon Dec 7 15:09:54 2015 -0600
--
hadoop-yarn-project/CHANGES.txt | 3 +++
.../ApplicationHistoryManagerOnTimelineStore.java | 7 +++
.../resourcemanager/metrics/AppAttemptFinishedEvent.java | 10 +-
.../resourcemanager/metrics/SystemMetricsPublisher.java | 6 +-
4 files changed, 24 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/22be47d7/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index c77d6b0..6171594 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -43,6 +43,9 @@ Release 2.7.3 - UNRELEASED
(with application having id > ) （Mohammad Shahid Khan & Varun Saxena
via jianhe)
+YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on
App page
+(Eric Payne via jeagles)
+
Release 2.7.2 - UNRELEASED
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/22be47d7/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
index 44d159c..6ab5b41 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java
@@ -418,6 +418,13 @@ public class ApplicationHistoryManagerOnTimelineStore
extends AbstractService
AppAttemptMetricsConstants.STATE_EVENT_INFO)
.toString());
}
+ if (eventInfo
+
.containsKey(AppAttemptMetricsConstants.MASTER_CONTAINER_EVENT_INFO)) {
+amContainerId =
+ConverterUtils.toContainerId(eventInfo.get(
+AppAttemptMetricsConstants.MASTER_CONTAINER_EVENT_INFO)
+.toString());
+ }
}
}
}
http://git-wip-us.apache.org/repos/asf/hadoop/blob/22be47d7/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
index 71d9363..fc1d10f 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/AppAttemptFinishedEvent.java
@@ -19,6 +19,7 @@
package org.apache.hadoop.yarn.server.resourcemana
hadoop git commit: YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 58d1df585 -> c293c5895
YARN-4183. Enabling generic application history forces every job to get a
timeline service delegation token (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/c293c589
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/c293c589
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/c293c589
Branch: refs/heads/trunk
Commit: c293c58954cdab25c8c69418b0e839883b563fa4
Parents: 58d1df5
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Thu Oct 29 16:41:10 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Thu Oct 29 16:41:10 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 3 +++
.../server/resourcemanager/metrics/SystemMetricsPublisher.java | 4 ++--
.../resourcemanager/metrics/TestSystemMetricsPublisher.java | 2 +-
3 files changed, 6 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c293c589/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index baa66c1..9fb2a9f 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -1138,6 +1138,9 @@ Release 2.7.2 - UNRELEASED
YARN-2902. Killing a container that is localizing can orphan resources in
the DOWNLOADING state (Varun Saxena via jlowe)
+YARN-4183. Enabling generic application history forces every job to get a
+timeline service delegation token (Mit Desai via jeagles)
+
Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c293c589/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
index 0f09735..c73e835 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
@@ -77,8 +77,8 @@ public class SystemMetricsPublisher extends CompositeService {
@Override
protected void serviceInit(Configuration conf) throws Exception {
publishSystemMetrics =
-conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED) &&
+conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED) &&
conf.getBoolean(YarnConfiguration.RM_SYSTEM_METRICS_PUBLISHER_ENABLED,
YarnConfiguration.DEFAULT_RM_SYSTEM_METRICS_PUBLISHER_ENABLED);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/c293c589/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
index 98daae7..e2fbf8d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
@@ -73,7 +73,7 @@ public class TestSystemMetricsPublisher {
@BeforeClass
public static void setup() throws Exception {
YarnConfiguration conf = new YarnConfiguration();
-conf.setBoolean(YarnConfiguration
hadoop git commit: YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 8943abfb0 -> 42bc525ac
YARN-4183. Enabling generic application history forces every job to get a
timeline service delegation token (jeagles)
(cherry picked from commit c293c58954cdab25c8c69418b0e839883b563fa4)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/42bc525a
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/42bc525a
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/42bc525a
Branch: refs/heads/branch-2
Commit: 42bc525acb78088ed952e92aa49408413937de86
Parents: 8943abf
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Thu Oct 29 16:41:10 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Thu Oct 29 16:42:49 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 3 +++
.../server/resourcemanager/metrics/SystemMetricsPublisher.java | 4 ++--
.../resourcemanager/metrics/TestSystemMetricsPublisher.java | 2 +-
3 files changed, 6 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/42bc525a/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index efe526c..cd92ee8 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -1086,6 +1086,9 @@ Release 2.7.2 - UNRELEASED
YARN-2902. Killing a container that is localizing can orphan resources in
the DOWNLOADING state (Varun Saxena via jlowe)
+YARN-4183. Enabling generic application history forces every job to get a
+timeline service delegation token (Mit Desai via jeagles)
+
Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/42bc525a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
index 0f09735..c73e835 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
@@ -77,8 +77,8 @@ public class SystemMetricsPublisher extends CompositeService {
@Override
protected void serviceInit(Configuration conf) throws Exception {
publishSystemMetrics =
-conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED) &&
+conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED) &&
conf.getBoolean(YarnConfiguration.RM_SYSTEM_METRICS_PUBLISHER_ENABLED,
YarnConfiguration.DEFAULT_RM_SYSTEM_METRICS_PUBLISHER_ENABLED);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/42bc525a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
index 98daae7..e2fbf8d 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
@@ -73,7 +73,7 @@ public class TestSystemMetricsPublisher {
@BeforeClass
public static void setup() throws Exception {
YarnConfigurati
hadoop git commit: YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.7 7499981d3 -> 3287a50cf
YARN-4183. Enabling generic application history forces every job to get a
timeline service delegation token (jeagles)
(cherry picked from commit c293c58954cdab25c8c69418b0e839883b563fa4)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3287a50c
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3287a50c
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3287a50c
Branch: refs/heads/branch-2.7
Commit: 3287a50cfe4b978d3776e3cce2218e0b10e3e519
Parents: 7499981
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Thu Oct 29 16:41:10 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Thu Oct 29 16:45:31 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 3 +++
.../server/resourcemanager/metrics/SystemMetricsPublisher.java | 4 ++--
.../resourcemanager/metrics/TestSystemMetricsPublisher.java | 2 +-
3 files changed, 6 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3287a50c/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 0421e41..94bdf7b 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -97,6 +97,9 @@ Release 2.7.2 - UNRELEASED
YARN-2902. Killing a container that is localizing can orphan resources in
the DOWNLOADING state (Varun Saxena via jlowe)
+YARN-4183. Enabling generic application history forces every job to get a
+timeline service delegation token (Mit Desai via jeagles)
+
Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3287a50c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
index b849b00..273c1d9 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
@@ -75,8 +75,8 @@ public class SystemMetricsPublisher extends CompositeService {
@Override
protected void serviceInit(Configuration conf) throws Exception {
publishSystemMetrics =
-conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED) &&
+conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
+YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED) &&
conf.getBoolean(YarnConfiguration.RM_SYSTEM_METRICS_PUBLISHER_ENABLED,
YarnConfiguration.DEFAULT_RM_SYSTEM_METRICS_PUBLISHER_ENABLED);
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3287a50c/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
index 7ed3835..40a4b9a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
@@ -66,7 +66,7 @@ public class TestSystemMetricsPublisher {
@BeforeClass
public static void setup() throws Exception {
YarnConfigurati
hadoop git commit: YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 35a303dfb -> f8adeb712
YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via
jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/f8adeb71
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/f8adeb71
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/f8adeb71
Branch: refs/heads/trunk
Commit: f8adeb712dc834c27cec15c04a986f2f635aba83
Parents: 35a303d
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Fri Oct 23 10:34:08 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Fri Oct 23 10:34:08 2015 -0500
--
.../HttpCrossOriginFilterInitializer.java | 74 +
.../hadoop/security/http/CrossOriginFilter.java | 259 +++
.../src/main/resources/core-default.xml | 36 ++
.../src/site/markdown/HttpAuthentication.md | 14 +
.../TestHttpCrossOriginFilterInitializer.java | 58
.../security/http/TestCrossOriginFilter.java| 330 +++
hadoop-yarn-project/CHANGES.txt | 2 +
.../hadoop/yarn/conf/YarnConfiguration.java | 10 +
.../src/main/resources/yarn-default.xml | 16 +
.../ApplicationHistoryServer.java | 14 +-
.../timeline/webapp/CrossOriginFilter.java | 259 ---
.../webapp/CrossOriginFilterInitializer.java| 40 +--
.../timeline/webapp/TestCrossOriginFilter.java | 324 --
.../TestCrossOriginFilterInitializer.java | 57
.../hadoop-yarn-server-common/pom.xml | 5 +
.../server/nodemanager/webapp/WebServer.java| 10 +-
.../server/resourcemanager/ResourceManager.java | 14 +-
.../src/site/markdown/NodeManagerRest.md| 10 +-
.../src/site/markdown/ResourceManagerRest.md| 8 +
19 files changed, 871 insertions(+), 669 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/f8adeb71/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
new file mode 100644
index 000..f9c1816
--- /dev/null
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.security;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.FilterContainer;
+import org.apache.hadoop.http.FilterInitializer;
+import org.apache.hadoop.security.http.CrossOriginFilter;
+
+public class HttpCrossOriginFilterInitializer extends FilterInitializer {
+
+ public static final String PREFIX = "hadoop.http.cross-origin.";
+ public static final String ENABLED_SUFFIX = "enabled";
+
+ private static final Log LOG =
+ LogFactory.getLog(HttpCrossOriginFilterInitializer.class);
+
+ @Override
+ public void initFilter(FilterContainer container, Configuration conf) {
+
+String key = getEnabledConfigKey();
+boolean enabled = conf.getBoolean(key, false);
+if (enabled) {
+ container.addGlobalFilter("Cross Origin Filter",
+ CrossOriginFilter.class.getName(),
+ getFilterParameters(conf, getPrefix()));
+} else {
+ LOG.info("CORS filter not enabled. Please set " + key
+ + " to 'true' to enable it");
+}
+ }
+
+ protected static Map<String, String> getFilterParameters(Configuration conf,
+ String prefix) {
+Map<String, String> filterParam
hadoop git commit: YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 4bb7e68eb -> 6db7bfbcf
YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via
jeagles)
(cherry picked from commit f8adeb712dc834c27cec15c04a986f2f635aba83)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/6db7bfbc
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/6db7bfbc
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/6db7bfbc
Branch: refs/heads/branch-2
Commit: 6db7bfbcfd853ffb1a752d2374d8e2d47444d800
Parents: 4bb7e68
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Fri Oct 23 10:34:08 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Fri Oct 23 10:43:59 2015 -0500
--
.../HttpCrossOriginFilterInitializer.java | 74 +
.../hadoop/security/http/CrossOriginFilter.java | 259 +++
.../src/main/resources/core-default.xml | 36 ++
.../src/site/markdown/HttpAuthentication.md | 14 +
.../TestHttpCrossOriginFilterInitializer.java | 58
.../security/http/TestCrossOriginFilter.java| 330 +++
hadoop-yarn-project/CHANGES.txt | 2 +
.../hadoop/yarn/conf/YarnConfiguration.java | 10 +
.../src/main/resources/yarn-default.xml | 16 +
.../ApplicationHistoryServer.java | 14 +-
.../timeline/webapp/CrossOriginFilter.java | 259 ---
.../webapp/CrossOriginFilterInitializer.java| 40 +--
.../timeline/webapp/TestCrossOriginFilter.java | 324 --
.../TestCrossOriginFilterInitializer.java | 57
.../hadoop-yarn-server-common/pom.xml | 5 +
.../server/nodemanager/webapp/WebServer.java| 10 +-
.../server/resourcemanager/ResourceManager.java | 14 +-
.../src/site/markdown/NodeManagerRest.md| 10 +-
.../src/site/markdown/ResourceManagerRest.md| 8 +
19 files changed, 871 insertions(+), 669 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/6db7bfbc/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
new file mode 100644
index 000..f9c1816
--- /dev/null
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.security;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.FilterContainer;
+import org.apache.hadoop.http.FilterInitializer;
+import org.apache.hadoop.security.http.CrossOriginFilter;
+
+public class HttpCrossOriginFilterInitializer extends FilterInitializer {
+
+ public static final String PREFIX = "hadoop.http.cross-origin.";
+ public static final String ENABLED_SUFFIX = "enabled";
+
+ private static final Log LOG =
+ LogFactory.getLog(HttpCrossOriginFilterInitializer.class);
+
+ @Override
+ public void initFilter(FilterContainer container, Configuration conf) {
+
+String key = getEnabledConfigKey();
+boolean enabled = conf.getBoolean(key, false);
+if (enabled) {
+ container.addGlobalFilter("Cross Origin Filter",
+ CrossOriginFilter.class.getName(),
+ getFilterParameters(conf, getPrefix()));
+} else {
+ LOG.info("CORS filter not enabled. Please set " + key
+ + " to 'true' to enable it");
+}
+ }
+
+ protected static Map<String, String> getFilterParameters(Configurati
hadoop git commit: YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.7 7021e015d -> 49a7d70f5
YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via
jeagles)
(cherry picked from commit f8adeb712dc834c27cec15c04a986f2f635aba83)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/49a7d70f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/49a7d70f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/49a7d70f
Branch: refs/heads/branch-2.7
Commit: 49a7d70f532086a8ba5a7c40873da656707d405a
Parents: 7021e01
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Fri Oct 23 10:34:08 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Fri Oct 23 11:18:32 2015 -0500
--
.../HttpCrossOriginFilterInitializer.java | 74 +
.../hadoop/security/http/CrossOriginFilter.java | 259 +++
.../src/main/resources/core-default.xml | 36 ++
.../src/site/markdown/HttpAuthentication.md | 14 +
.../TestHttpCrossOriginFilterInitializer.java | 58
.../security/http/TestCrossOriginFilter.java| 330 +++
hadoop-yarn-project/CHANGES.txt | 2 +
.../hadoop/yarn/conf/YarnConfiguration.java | 12 +-
.../src/main/resources/yarn-default.xml | 16 +
.../ApplicationHistoryServer.java | 14 +-
.../timeline/webapp/CrossOriginFilter.java | 259 ---
.../webapp/CrossOriginFilterInitializer.java| 40 +--
.../timeline/webapp/TestCrossOriginFilter.java | 324 --
.../TestCrossOriginFilterInitializer.java | 57
.../hadoop-yarn-server-common/pom.xml | 5 +
.../server/nodemanager/webapp/WebServer.java| 10 +-
.../server/resourcemanager/ResourceManager.java | 14 +-
.../src/site/markdown/NodeManagerRest.md| 10 +-
.../src/site/markdown/ResourceManagerRest.md| 8 +
19 files changed, 872 insertions(+), 670 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/49a7d70f/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
new file mode 100644
index 000..f9c1816
--- /dev/null
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/HttpCrossOriginFilterInitializer.java
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.security;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.http.FilterContainer;
+import org.apache.hadoop.http.FilterInitializer;
+import org.apache.hadoop.security.http.CrossOriginFilter;
+
+public class HttpCrossOriginFilterInitializer extends FilterInitializer {
+
+ public static final String PREFIX = "hadoop.http.cross-origin.";
+ public static final String ENABLED_SUFFIX = "enabled";
+
+ private static final Log LOG =
+ LogFactory.getLog(HttpCrossOriginFilterInitializer.class);
+
+ @Override
+ public void initFilter(FilterContainer container, Configuration conf) {
+
+String key = getEnabledConfigKey();
+boolean enabled = conf.getBoolean(key, false);
+if (enabled) {
+ container.addGlobalFilter("Cross Origin Filter",
+ CrossOriginFilter.class.getName(),
+ getFilterParameters(conf, getPrefix()));
+} else {
+ LOG.info("CORS filter not enabled. Please set " + key
+ + " to 'true' to enable it");
+}
+ }
+
+ protected static Map<String, String> getFilterParameters(Conf
hadoop git commit: YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 39e7548cd -> cc8093e5a
YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
(cherry picked from commit 25f8f801d15e3d9f27f4a2a198262407203e14a5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/cc8093e5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/cc8093e5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/cc8093e5
Branch: refs/heads/branch-2
Commit: cc8093e5acaaae3932be7f5c7c8681ed2415821b
Parents: 39e7548
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed Oct 21 15:38:08 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed Oct 21 15:38:48 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 2 +
.../hadoop/yarn/conf/YarnConfiguration.java | 17 +++
.../org/apache/hadoop/yarn/webapp/WebApps.java | 18 +--
.../src/main/resources/yarn-default.xml | 6 +++
.../ApplicationHistoryServer.java | 44 ++--
.../TestApplicationHistoryServer.java | 53 +++-
.../src/site/markdown/TimelineServer.md | 8 +++
7 files changed, 140 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/cc8093e5/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 5d4945a..5bc144d 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -937,6 +937,8 @@ Release 2.7.2 - UNRELEASED
YARN-2801. Add documentation for node labels feature. (Wangda Tan and
Naganarasimha
G R via ozawa)
+YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/cc8093e5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 60c776c..0fcdce2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1524,6 +1524,23 @@ public class YarnConfiguration extends Configuration {
public static final String TIMELINE_SERVICE_PREFIX =
YARN_PREFIX + "timeline-service.";
+ /**
+ * Comma seperated list of names for UIs hosted in the timeline server
+ * (For pluggable UIs).
+ */
+ public static final String TIMELINE_SERVICE_UI_NAMES =
+ TIMELINE_SERVICE_PREFIX + "ui-names";
+
+ /** Relative web path that will serve up this UI (For pluggable UIs). */
+ public static final String TIMELINE_SERVICE_UI_WEB_PATH_PREFIX =
+ TIMELINE_SERVICE_PREFIX + "ui-web-path.";
+
+ /**
+ * Path to war file or static content directory for this UI
+ * (For pluggable UIs).
+ */
+ public static final String TIMELINE_SERVICE_UI_ON_DISK_PATH_PREFIX =
+ TIMELINE_SERVICE_PREFIX + "ui-on-disk-path.";
// mark app-history related configs @Private as application history is going
// to be integrated into the timeline service
http://git-wip-us.apache.org/repos/asf/hadoop/blob/cc8093e5/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
index bda24aa..0c6edad 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
@@ -166,7 +166,7 @@ public class WebApps {
return this;
}
-public WebApp start(WebApp webapp) {
+public WebApp build(WebApp webapp) {
if (webapp == null) {
webapp = new WebApp() {
@Override
@@ -271,8 +271,7 @@ public class WebApps {
webapp.setConf(conf);
webapp.setHttpServer(server);
-server.start();
-LOG.info("Web app /"+ name +" started at "+
server.ge
hadoop git commit: YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk d759b4bd7 -> 25f8f801d
YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/25f8f801
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/25f8f801
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/25f8f801
Branch: refs/heads/trunk
Commit: 25f8f801d15e3d9f27f4a2a198262407203e14a5
Parents: d759b4b
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed Oct 21 15:38:08 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed Oct 21 15:38:08 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 2 +
.../hadoop/yarn/conf/YarnConfiguration.java | 17 +++
.../org/apache/hadoop/yarn/webapp/WebApps.java | 18 +--
.../src/main/resources/yarn-default.xml | 6 +++
.../ApplicationHistoryServer.java | 44 ++--
.../TestApplicationHistoryServer.java | 53 +++-
.../src/site/markdown/TimelineServer.md | 8 +++
7 files changed, 140 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/25f8f801/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 8277e05..ae26386 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -989,6 +989,8 @@ Release 2.7.2 - UNRELEASED
YARN-2801. Add documentation for node labels feature. (Wangda Tan and
Naganarasimha
G R via ozawa)
+YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/25f8f801/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 55b829f..3e89259 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1524,6 +1524,23 @@ public class YarnConfiguration extends Configuration {
public static final String TIMELINE_SERVICE_PREFIX =
YARN_PREFIX + "timeline-service.";
+ /**
+ * Comma seperated list of names for UIs hosted in the timeline server
+ * (For pluggable UIs).
+ */
+ public static final String TIMELINE_SERVICE_UI_NAMES =
+ TIMELINE_SERVICE_PREFIX + "ui-names";
+
+ /** Relative web path that will serve up this UI (For pluggable UIs). */
+ public static final String TIMELINE_SERVICE_UI_WEB_PATH_PREFIX =
+ TIMELINE_SERVICE_PREFIX + "ui-web-path.";
+
+ /**
+ * Path to war file or static content directory for this UI
+ * (For pluggable UIs).
+ */
+ public static final String TIMELINE_SERVICE_UI_ON_DISK_PATH_PREFIX =
+ TIMELINE_SERVICE_PREFIX + "ui-on-disk-path.";
// mark app-history related configs @Private as application history is going
// to be integrated into the timeline service
http://git-wip-us.apache.org/repos/asf/hadoop/blob/25f8f801/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
index bda24aa..0c6edad 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
@@ -166,7 +166,7 @@ public class WebApps {
return this;
}
-public WebApp start(WebApp webapp) {
+public WebApp build(WebApp webapp) {
if (webapp == null) {
webapp = new WebApp() {
@Override
@@ -271,8 +271,7 @@ public class WebApps {
webapp.setConf(conf);
webapp.setHttpServer(server);
-server.start();
-LOG.info("Web app /"+ name +" started at "+
server.getConnectorAddress(0).getPort());
+
} catch (ClassNotFoundException e) {
hadoop git commit: YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.7 c74a32ea4 -> 7d3d0d605
YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
(cherry picked from commit 25f8f801d15e3d9f27f4a2a198262407203e14a5)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7d3d0d60
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7d3d0d60
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7d3d0d60
Branch: refs/heads/branch-2.7
Commit: 7d3d0d605238cf0191eef988d3bfc5d2008a4241
Parents: c74a32e
Author: Jonathan Eagles <jeag...@yahoo-inc.com>
Authored: Wed Oct 21 15:38:08 2015 -0500
Committer: Jonathan Eagles <jeag...@yahoo-inc.com>
Committed: Wed Oct 21 15:40:24 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 2 +
.../hadoop/yarn/conf/YarnConfiguration.java | 17 +++
.../org/apache/hadoop/yarn/webapp/WebApps.java | 18 +--
.../src/main/resources/yarn-default.xml | 6 +++
.../ApplicationHistoryServer.java | 44 ++--
.../TestApplicationHistoryServer.java | 53 +++-
.../src/site/markdown/TimelineServer.md | 8 +++
7 files changed, 140 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7d3d0d60/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 3d39e30..701d1e1 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -17,6 +17,8 @@ Release 2.7.2 - UNRELEASED
YARN-2801. Add documentation for node labels feature. (Wangda Tan and
Naganarasimha
G R via ozawa)
+YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7d3d0d60/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index c029d90..0908e65 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1307,6 +1307,23 @@ public class YarnConfiguration extends Configuration {
public static final String TIMELINE_SERVICE_PREFIX =
YARN_PREFIX + "timeline-service.";
+ /**
+ * Comma seperated list of names for UIs hosted in the timeline server
+ * (For pluggable UIs).
+ */
+ public static final String TIMELINE_SERVICE_UI_NAMES =
+ TIMELINE_SERVICE_PREFIX + "ui-names";
+
+ /** Relative web path that will serve up this UI (For pluggable UIs). */
+ public static final String TIMELINE_SERVICE_UI_WEB_PATH_PREFIX =
+ TIMELINE_SERVICE_PREFIX + "ui-web-path.";
+
+ /**
+ * Path to war file or static content directory for this UI
+ * (For pluggable UIs).
+ */
+ public static final String TIMELINE_SERVICE_UI_ON_DISK_PATH_PREFIX =
+ TIMELINE_SERVICE_PREFIX + "ui-on-disk-path.";
// mark app-history related configs @Private as application history is going
// to be integrated into the timeline service
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7d3d0d60/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
index bda24aa..0c6edad 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java
@@ -166,7 +166,7 @@ public class WebApps {
return this;
}
-public WebApp start(WebApp webapp) {
+public WebApp build(WebApp webapp) {
if (webapp == null) {
webapp = new WebApp() {
@Override
@@ -271,8 +271,7 @@ public class WebApps {
webapp.setConf(conf);
webapp.setHttpServer(server);
-server.start();
-LOG.info("Web app /"+ name +" started at "+
server.ge
hadoop git commit: YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 c6e79178d - 899df5bce
YARN-3978. Configurably turn off the saving of container info in Generic AHS
(Eric Payne via jeagles)
(cherry picked from commit 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/899df5bc
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/899df5bc
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/899df5bc
Branch: refs/heads/branch-2
Commit: 899df5bce03ea4f94487e48c1d38bd30ae10c26f
Parents: c6e7917
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Mon Aug 3 10:38:05 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Mon Aug 3 10:40:04 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 3 +
.../hadoop/yarn/conf/YarnConfiguration.java | 9 +++
.../hadoop/yarn/server/webapp/AppBlock.java | 12 +--
.../rmcontainer/RMContainerImpl.java| 43 ++-
.../resourcemanager/TestClientRMService.java| 1 +
.../rmcontainer/TestRMContainerImpl.java| 79 +++-
.../scheduler/capacity/TestChildQueueOrder.java | 1 +
.../scheduler/capacity/TestLeafQueue.java | 2 +
.../scheduler/capacity/TestReservations.java| 6 +-
.../scheduler/fifo/TestFifoScheduler.java | 2 +
10 files changed, 146 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/899df5bc/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 79a3f1b..65358a5 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -681,6 +681,9 @@ Release 2.7.2 - UNRELEASED
YARN-3967. Fetch the application report from the AHS if the RM does not
know about it.
(Mit Desai via xgong)
+YARN-3978. Configurably turn off the saving of container info in Generic
AHS
+(Eric Payne via jeagles)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/899df5bc/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 7d602a6..48b7ccd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1435,6 +1435,15 @@ public class YarnConfiguration extends Configuration {
public static final String APPLICATION_HISTORY_STORE =
APPLICATION_HISTORY_PREFIX + store-class;
+ /** Save container meta-info in the application history store. */
+ @Private
+ public static final String
+ APPLICATION_HISTORY_SAVE_NON_AM_CONTAINER_META_INFO =
+APPLICATION_HISTORY_PREFIX + save-non-am-container-meta-info;
+ @Private
+ public static final boolean
+DEFAULT_APPLICATION_HISTORY_SAVE_NON_AM_CONTAINER_META_INFO = true;
+
/** URI for FileSystemApplicationHistoryStore */
@Private
public static final String FS_APPLICATION_HISTORY_STORE_URI =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/899df5bc/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
index eec32b2..871bac3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
@@ -266,11 +266,13 @@ public class AppBlock extends HtmlBlock {
@Override
public ContainerReport run() throws Exception {
ContainerReport report = null;
- try {
-report = appBaseProt.getContainerReport(request)
-.getContainerReport();
- } catch (ContainerNotFoundException ex
hadoop git commit: YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.7 e1ae722a7 - dcffe4341
YARN-3978. Configurably turn off the saving of container info in Generic AHS
(Eric Payne via jeagles)
(cherry picked from commit 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dcffe434
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dcffe434
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dcffe434
Branch: refs/heads/branch-2.7
Commit: dcffe4341947f396ff3ba196db03ba921251a650
Parents: e1ae722
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Mon Aug 3 10:38:05 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Mon Aug 3 10:53:43 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 3 +
.../hadoop/yarn/conf/YarnConfiguration.java | 9 +++
.../hadoop/yarn/server/webapp/AppBlock.java | 12 +--
.../rmcontainer/RMContainerImpl.java| 43 ++-
.../resourcemanager/TestClientRMService.java| 1 +
.../rmcontainer/TestRMContainerImpl.java| 79 +++-
.../scheduler/capacity/TestChildQueueOrder.java | 1 +
.../scheduler/capacity/TestLeafQueue.java | 5 ++
.../scheduler/capacity/TestReservations.java| 6 +-
.../scheduler/fifo/TestFifoScheduler.java | 2 +
10 files changed, 149 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/dcffe434/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 738f6d8..ce9f26c 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -14,6 +14,9 @@ Release 2.7.2 - UNRELEASED
YARN-3967. Fetch the application report from the AHS if the RM does not
know about it.
(Mit Desai via xgong)
+YARN-3978. Configurably turn off the saving of container info in Generic
AHS
+(Eric Payne via jeagles)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/dcffe434/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index da076eb..9953d40 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1311,6 +1311,15 @@ public class YarnConfiguration extends Configuration {
public static final String APPLICATION_HISTORY_STORE =
APPLICATION_HISTORY_PREFIX + store-class;
+ /** Save container meta-info in the application history store. */
+ @Private
+ public static final String
+ APPLICATION_HISTORY_SAVE_NON_AM_CONTAINER_META_INFO =
+APPLICATION_HISTORY_PREFIX + save-non-am-container-meta-info;
+ @Private
+ public static final boolean
+DEFAULT_APPLICATION_HISTORY_SAVE_NON_AM_CONTAINER_META_INFO = true;
+
/** URI for FileSystemApplicationHistoryStore */
@Private
public static final String FS_APPLICATION_HISTORY_STORE_URI =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/dcffe434/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
index 140d5ea..c620f8e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
hadoop git commit: YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 90b510405 - 3cd02b952
YARN-3978. Configurably turn off the saving of container info in Generic AHS
(Eric Payne via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3cd02b95
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3cd02b95
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3cd02b95
Branch: refs/heads/trunk
Commit: 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d
Parents: 90b5104
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Mon Aug 3 10:38:05 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Mon Aug 3 10:38:05 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 3 +
.../hadoop/yarn/conf/YarnConfiguration.java | 9 +++
.../hadoop/yarn/server/webapp/AppBlock.java | 12 +--
.../rmcontainer/RMContainerImpl.java| 43 ++-
.../resourcemanager/TestClientRMService.java| 1 +
.../rmcontainer/TestRMContainerImpl.java| 79 +++-
.../scheduler/capacity/TestChildQueueOrder.java | 1 +
.../scheduler/capacity/TestLeafQueue.java | 2 +
.../scheduler/capacity/TestReservations.java| 6 +-
.../scheduler/fifo/TestFifoScheduler.java | 2 +
10 files changed, 146 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3cd02b95/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 61b3cce..cd8cbd3 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -733,6 +733,9 @@ Release 2.7.2 - UNRELEASED
YARN-3967. Fetch the application report from the AHS if the RM does not
know about it.
(Mit Desai via xgong)
+YARN-3978. Configurably turn off the saving of container info in Generic
AHS
+(Eric Payne via jeagles)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3cd02b95/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
index 9832729..f1baf5c 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
@@ -1435,6 +1435,15 @@ public class YarnConfiguration extends Configuration {
public static final String APPLICATION_HISTORY_STORE =
APPLICATION_HISTORY_PREFIX + store-class;
+ /** Save container meta-info in the application history store. */
+ @Private
+ public static final String
+ APPLICATION_HISTORY_SAVE_NON_AM_CONTAINER_META_INFO =
+APPLICATION_HISTORY_PREFIX + save-non-am-container-meta-info;
+ @Private
+ public static final boolean
+DEFAULT_APPLICATION_HISTORY_SAVE_NON_AM_CONTAINER_META_INFO = true;
+
/** URI for FileSystemApplicationHistoryStore */
@Private
public static final String FS_APPLICATION_HISTORY_STORE_URI =
http://git-wip-us.apache.org/repos/asf/hadoop/blob/3cd02b95/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
index eec32b2..871bac3 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
@@ -266,11 +266,13 @@ public class AppBlock extends HtmlBlock {
@Override
public ContainerReport run() throws Exception {
ContainerReport report = null;
- try {
-report = appBaseProt.getContainerReport(request)
-.getContainerReport();
- } catch (ContainerNotFoundException ex) {
-LOG.warn(ex.getMessage());
+ if (request.getContainerId() != null
hadoop git commit: YARN-3905. Application History Server UI NPEs when accessing apps run after RM restart (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2.7 2fe313661 - 009f5fbad
YARN-3905. Application History Server UI NPEs when accessing apps run after RM
restart (Eric Payne via jeagles)
(cherry picked from commit 7faae0e6fe027a3886d9f4e290b6a488a2c55b3a)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/009f5fba
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/009f5fba
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/009f5fba
Branch: refs/heads/branch-2.7
Commit: 009f5fbad65ba2e9cc6c49ba196ee79cbbe59541
Parents: 2fe3136
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Fri Jul 17 11:02:11 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Fri Jul 17 11:16:37 2015 -0500
--
hadoop-yarn-project/CHANGES.txt| 3 +++
.../java/org/apache/hadoop/yarn/server/webapp/AppBlock.java| 6 ++
2 files changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/009f5fba/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index d87a222..cffd2d4 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -23,6 +23,9 @@ Release 2.7.2 - UNRELEASED
YARN-3690. [JDK8] 'mvn site' fails. (Brahma Reddy Battula via aajisaka)
+YARN-3905. Application History Server UI NPEs when accessing apps run after
+RM restart (Eric Payne via jeagles)
+
Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/009f5fba/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
index e635c9f..140d5ea 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
@@ -36,7 +36,6 @@ import
org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest;
import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
-import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.YarnApplicationState;
@@ -227,10 +226,9 @@ public class AppBlock extends HtmlBlock {
AppAttemptInfo appAttempt = new AppAttemptInfo(appAttemptReport);
ContainerReport containerReport = null;
try {
-// AM container is always the first container of the attempt
final GetContainerReportRequest request =
-GetContainerReportRequest.newInstance(ContainerId.newContainerId(
- appAttemptReport.getApplicationAttemptId(), 1));
+GetContainerReportRequest.newInstance(
+ appAttemptReport.getAMContainerId());
if (callerUGI == null) {
containerReport =
appBaseProt.getContainerReport(request).getContainerReport();
hadoop git commit: YARN-3905. Application History Server UI NPEs when accessing apps run after RM restart (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 8018041b4 - b28993951
YARN-3905. Application History Server UI NPEs when accessing apps run after RM
restart (Eric Payne via jeagles)
(cherry picked from commit 7faae0e6fe027a3886d9f4e290b6a488a2c55b3a)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/b2899395
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/b2899395
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/b2899395
Branch: refs/heads/branch-2
Commit: b2899395142848e338aee970114430cc125970f2
Parents: 8018041
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Fri Jul 17 11:02:11 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Fri Jul 17 11:03:34 2015 -0500
--
hadoop-yarn-project/CHANGES.txt| 3 +++
.../java/org/apache/hadoop/yarn/server/webapp/AppBlock.java| 6 ++
2 files changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b2899395/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 12483f2..6ee1215 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -621,6 +621,9 @@ Release 2.7.2 - UNRELEASED
YARN-3690. [JDK8] 'mvn site' fails. (Brahma Reddy Battula via aajisaka)
+YARN-3905. Application History Server UI NPEs when accessing apps run after
+RM restart (Eric Payne via jeagles)
+
Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b2899395/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
index f46197e..eec32b2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
@@ -38,7 +38,6 @@ import
org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest;
import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
-import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.LogAggregationStatus;
@@ -255,10 +254,9 @@ public class AppBlock extends HtmlBlock {
AppAttemptInfo appAttempt = new AppAttemptInfo(appAttemptReport);
ContainerReport containerReport;
try {
-// AM container is always the first container of the attempt
final GetContainerReportRequest request =
-GetContainerReportRequest.newInstance(ContainerId.newContainerId(
- appAttemptReport.getApplicationAttemptId(), 1));
+GetContainerReportRequest.newInstance(
+ appAttemptReport.getAMContainerId());
if (callerUGI == null) {
containerReport =
appBaseProt.getContainerReport(request).getContainerReport();
hadoop git commit: YARN-3905. Application History Server UI NPEs when accessing apps run after RM restart (Eric Payne via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 9b272ccae - 7faae0e6f
YARN-3905. Application History Server UI NPEs when accessing apps run after RM
restart (Eric Payne via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7faae0e6
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7faae0e6
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7faae0e6
Branch: refs/heads/trunk
Commit: 7faae0e6fe027a3886d9f4e290b6a488a2c55b3a
Parents: 9b272cc
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Fri Jul 17 11:02:11 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Fri Jul 17 11:02:11 2015 -0500
--
hadoop-yarn-project/CHANGES.txt| 3 +++
.../java/org/apache/hadoop/yarn/server/webapp/AppBlock.java| 6 ++
2 files changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7faae0e6/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index 7a94e09..df27023 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -669,6 +669,9 @@ Release 2.7.2 - UNRELEASED
YARN-3690. [JDK8] 'mvn site' fails. (Brahma Reddy Battula via aajisaka)
+YARN-3905. Application History Server UI NPEs when accessing apps run after
+RM restart (Eric Payne via jeagles)
+
Release 2.7.1 - 2015-07-06
INCOMPATIBLE CHANGES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7faae0e6/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
index f46197e..eec32b2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppBlock.java
@@ -38,7 +38,6 @@ import
org.apache.hadoop.yarn.api.protocolrecords.GetContainerReportRequest;
import org.apache.hadoop.yarn.api.records.ApplicationAttemptReport;
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
-import org.apache.hadoop.yarn.api.records.ContainerId;
import org.apache.hadoop.yarn.api.records.ContainerReport;
import org.apache.hadoop.yarn.api.records.FinalApplicationStatus;
import org.apache.hadoop.yarn.api.records.LogAggregationStatus;
@@ -255,10 +254,9 @@ public class AppBlock extends HtmlBlock {
AppAttemptInfo appAttempt = new AppAttemptInfo(appAttemptReport);
ContainerReport containerReport;
try {
-// AM container is always the first container of the attempt
final GetContainerReportRequest request =
-GetContainerReportRequest.newInstance(ContainerId.newContainerId(
- appAttemptReport.getApplicationAttemptId(), 1));
+GetContainerReportRequest.newInstance(
+ appAttemptReport.getAMContainerId());
if (callerUGI == null) {
containerReport =
appBaseProt.getContainerReport(request).getContainerReport();
hadoop git commit: YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 2236b577a - 4c659ddbf
YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/4c659ddb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/4c659ddb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/4c659ddb
Branch: refs/heads/trunk
Commit: 4c659ddbf7629aae92e66a5b54893e9c1c68dfb0
Parents: 2236b57
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Wed Jun 24 16:16:52 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Wed Jun 24 16:16:52 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 2 +
.../server/timeline/TimelineDataManager.java| 123 +
.../timeline/TimelineDataManagerMetrics.java| 174 +++
.../TestApplicationHistoryClientService.java| 1 +
...pplicationHistoryManagerOnTimelineStore.java | 1 +
.../webapp/TestAHSWebServices.java | 1 +
.../timeline/TestTimelineDataManager.java | 1 +
7 files changed, 303 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4c659ddb/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index f831f0a..9e70ec9 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -60,6 +60,8 @@ Release 2.8.0 - UNRELEASED
NEW FEATURES
+YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
+
YARN-3345. Add non-exclusive node label API. (Wangda Tan via jianhe)
YARN-3365. Enhanced NodeManager to support using the 'tc' tool via
http://git-wip-us.apache.org/repos/asf/hadoop/blob/4c659ddb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
index 86aae77..459fd4e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
@@ -31,6 +31,7 @@ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.service.AbstractService;
+import org.apache.hadoop.util.Time;
import org.apache.hadoop.yarn.api.records.ApplicationAccessType;
import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
@@ -56,6 +57,7 @@ public class TimelineDataManager extends AbstractService {
@VisibleForTesting
public static final String DEFAULT_DOMAIN_ID = DEFAULT;
+ private TimelineDataManagerMetrics metrics;
private TimelineStore store;
private TimelineACLsManager timelineACLsManager;
@@ -69,6 +71,7 @@ public class TimelineDataManager extends AbstractService {
@Override
protected void serviceInit(Configuration conf) throws Exception {
+metrics = TimelineDataManagerMetrics.create();
TimelineDomain domain = store.getDomain(DEFAULT);
// it is okay to reuse an existing domain even if it was created by another
// user of the timeline server before, because it allows everybody to
access.
@@ -130,6 +133,38 @@ public class TimelineDataManager extends AbstractService {
Long limit,
EnumSetField fields,
UserGroupInformation callerUGI) throws YarnException, IOException {
+long startTime = Time.monotonicNow();
+metrics.incrGetEntitiesOps();
+try {
+ TimelineEntities entities = doGetEntities(
+ entityType,
+ primaryFilter,
+ secondaryFilter,
+ windowStart,
+ windowEnd,
+ fromId,
+ fromTs,
+ limit,
+ fields,
+ callerUGI);
+ metrics.incrGetEntitiesTotal(entities.getEntities().size());
+ return entities;
+} finally {
+ metrics.addGetEntitiesTime(Time.monotonicNow() - startTime);
+}
+ }
+
+ private TimelineEntities doGetEntities
hadoop git commit: YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 bb6e28592 - d1f0ff165
YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
(cherry picked from commit 4c659ddbf7629aae92e66a5b54893e9c1c68dfb0)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/d1f0ff16
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/d1f0ff16
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/d1f0ff16
Branch: refs/heads/branch-2
Commit: d1f0ff165b11879212e579aeba138b04ef165a55
Parents: bb6e285
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Wed Jun 24 16:16:52 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Wed Jun 24 16:17:32 2015 -0500
--
hadoop-yarn-project/CHANGES.txt | 2 +
.../server/timeline/TimelineDataManager.java| 123 +
.../timeline/TimelineDataManagerMetrics.java| 174 +++
.../TestApplicationHistoryClientService.java| 1 +
...pplicationHistoryManagerOnTimelineStore.java | 1 +
.../webapp/TestAHSWebServices.java | 1 +
.../timeline/TestTimelineDataManager.java | 1 +
7 files changed, 303 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d1f0ff16/hadoop-yarn-project/CHANGES.txt
--
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index a0b08e7..2fb5075 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -9,6 +9,8 @@ Release 2.8.0 - UNRELEASED
NEW FEATURES
+YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles)
+
YARN-3345. Add non-exclusive node label API. (Wangda Tan via jianhe)
YARN-3365. Enhanced NodeManager to support using the 'tc' tool via
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d1f0ff16/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
--
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
index 86aae77..459fd4e 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/TimelineDataManager.java
@@ -31,6 +31,7 @@ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.service.AbstractService;
+import org.apache.hadoop.util.Time;
import org.apache.hadoop.yarn.api.records.ApplicationAccessType;
import org.apache.hadoop.yarn.api.records.timeline.TimelineEntities;
import org.apache.hadoop.yarn.api.records.timeline.TimelineEntity;
@@ -56,6 +57,7 @@ public class TimelineDataManager extends AbstractService {
@VisibleForTesting
public static final String DEFAULT_DOMAIN_ID = DEFAULT;
+ private TimelineDataManagerMetrics metrics;
private TimelineStore store;
private TimelineACLsManager timelineACLsManager;
@@ -69,6 +71,7 @@ public class TimelineDataManager extends AbstractService {
@Override
protected void serviceInit(Configuration conf) throws Exception {
+metrics = TimelineDataManagerMetrics.create();
TimelineDomain domain = store.getDomain(DEFAULT);
// it is okay to reuse an existing domain even if it was created by another
// user of the timeline server before, because it allows everybody to
access.
@@ -130,6 +133,38 @@ public class TimelineDataManager extends AbstractService {
Long limit,
EnumSetField fields,
UserGroupInformation callerUGI) throws YarnException, IOException {
+long startTime = Time.monotonicNow();
+metrics.incrGetEntitiesOps();
+try {
+ TimelineEntities entities = doGetEntities(
+ entityType,
+ primaryFilter,
+ secondaryFilter,
+ windowStart,
+ windowEnd,
+ fromId,
+ fromTs,
+ limit,
+ fields,
+ callerUGI);
+ metrics.incrGetEntitiesTotal(entities.getEntities().size());
+ return entities;
+} finally {
+ metrics.addGetEntitiesTime(Time.monotonicNow
hadoop git commit: HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error occurs (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 2df00d53d - 2236b577a
HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error occurs
(Jason Lowe via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2236b577
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2236b577
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2236b577
Branch: refs/heads/trunk
Commit: 2236b577a34b069c0d1f91da99f63a199f260ac2
Parents: 2df00d5
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Wed Jun 24 14:51:04 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Wed Jun 24 14:51:04 2015 -0500
--
hadoop-common-project/hadoop-common/CHANGES.txt| 2 ++
.../java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java| 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2236b577/hadoop-common-project/hadoop-common/CHANGES.txt
--
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt
b/hadoop-common-project/hadoop-common/CHANGES.txt
index e1d9ca9..b2975dc 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -654,6 +654,8 @@ Release 2.8.0 - UNRELEASED
HADOOP-7139. Allow appending to existing SequenceFiles
(kanaka kumar avvaru via vinayakumarb)
+HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error
+occurs (Jason Lowe via jeagles)
OPTIMIZATIONS
HADOOP-11785. Reduce the number of listStatus operation in distcp
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2236b577/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
index a94d814..a1d258d 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
@@ -367,7 +367,7 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
try {
onTimerEvent();
} catch (Exception e) {
- LOG.warn(e);
+ LOG.warn(Error invoking metrics timer, e);
}
}
}, millis, millis);
hadoop git commit: HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error occurs (Jason Lowe via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 1619d8d62 - bb6e28592
HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error occurs
(Jason Lowe via jeagles)
(cherry picked from commit 2236b577a34b069c0d1f91da99f63a199f260ac2)
Conflicts:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bb6e2859
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bb6e2859
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bb6e2859
Branch: refs/heads/branch-2
Commit: bb6e28592e1bfeaf1e9648d51a67b469af1472b1
Parents: 1619d8d
Author: Jonathan Eagles jeag...@yahoo-inc.com
Authored: Wed Jun 24 14:51:04 2015 -0500
Committer: Jonathan Eagles jeag...@yahoo-inc.com
Committed: Wed Jun 24 14:54:11 2015 -0500
--
hadoop-common-project/hadoop-common/CHANGES.txt | 2 ++
.../java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java | 5 ++---
2 files changed, 4 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/bb6e2859/hadoop-common-project/hadoop-common/CHANGES.txt
--
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt
b/hadoop-common-project/hadoop-common/CHANGES.txt
index 894aa94..ed3e5f0 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -161,6 +161,8 @@ Release 2.8.0 - UNRELEASED
HADOOP-7139. Allow appending to existing SequenceFiles
(kanaka kumar avvaru via vinayakumarb)
+HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error
+occurs (Jason Lowe via jeagles)
OPTIMIZATIONS
HADOOP-11785. Reduce the number of listStatus operation in distcp
http://git-wip-us.apache.org/repos/asf/hadoop/blob/bb6e2859/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
--
diff --git
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
index b7f264b..fb2aa0e 100644
---
a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
+++
b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/impl/MetricsSystemImpl.java
@@ -368,9 +368,8 @@ public class MetricsSystemImpl extends MetricsSystem
implements MetricsSource {
public void run() {
try {
onTimerEvent();
-}
-catch (Exception e) {
- LOG.warn(e);
+} catch (Exception e) {
+ LOG.warn(Error invoking metrics timer, e);
}
}
}, millis, millis);
hadoop git commit: HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles)
Repository: hadoop Updated Branches: refs/heads/trunk 279958b77 - 2d7363b27 HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/2d7363b2 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/2d7363b2 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/2d7363b2 Branch: refs/heads/trunk Commit: 2d7363b27360e36fdd62546c0f9d0b1d78133f29 Parents: 279958b Author: Jonathan Eagles jeag...@gmail.com Authored: Fri May 1 13:11:50 2015 -0500 Committer: Jonathan Eagles jeag...@gmail.com Committed: Fri May 1 13:11:50 2015 -0500 -- hadoop-build-tools/pom.xml | 28 +++ .../checkstyle/checkstyle-noframes-sorted.xsl | 194 +++ .../main/resources/checkstyle/checkstyle.xml| 186 ++ .../hadoop-common/dev-support/checkstyle.xml| 185 -- .../src/test/checkstyle-noframes-sorted.xsl | 194 --- .../dev-support/checkstyle-noframes-sorted.xsl | 178 - .../hadoop-hdfs/dev-support/checkstyle.xml | 169 hadoop-project-dist/pom.xml | 11 -- hadoop-project/pom.xml | 6 - .../hadoop-azure/src/config/checkstyle.xml | 1 - hadoop-tools/hadoop-distcp/pom.xml | 7 - pom.xml | 32 +++ 12 files changed, 440 insertions(+), 751 deletions(-) -- http://git-wip-us.apache.org/repos/asf/hadoop/blob/2d7363b2/hadoop-build-tools/pom.xml -- diff --git a/hadoop-build-tools/pom.xml b/hadoop-build-tools/pom.xml new file mode 100644 index 000..1931072 --- /dev/null +++ b/hadoop-build-tools/pom.xml @@ -0,0 +1,28 @@ +?xml version=1.0 encoding=UTF-8? +!-- + Licensed under the Apache License, Version 2.0 (the License); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an AS IS BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +-- +project xmlns=http://maven.apache.org/POM/4.0.0; + xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation=http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd; +parent + artifactIdhadoop-main/artifactId + groupIdorg.apache.hadoop/groupId + version3.0.0-SNAPSHOT/version +/parent +modelVersion4.0.0/modelVersion + +artifactIdhadoop-build-tools/artifactId + descriptionApache Hadoop Build Tools Project/description + nameApache Hadoop Build Tools/name +/project \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hadoop/blob/2d7363b2/hadoop-build-tools/src/main/resources/checkstyle/checkstyle-noframes-sorted.xsl -- diff --git a/hadoop-build-tools/src/main/resources/checkstyle/checkstyle-noframes-sorted.xsl b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle-noframes-sorted.xsl new file mode 100644 index 000..b7826e3 --- /dev/null +++ b/hadoop-build-tools/src/main/resources/checkstyle/checkstyle-noframes-sorted.xsl @@ -0,0 +1,194 @@ +!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the License); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an AS IS BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +-- +xsl:stylesheetxmlns:xsl=http://www.w3.org/1999/XSL/Transform; version=1.0 +xsl:output method=html indent=yes/ +xsl:decimal-format decimal-separator=. grouping-separator=, / + +xsl:key name=files match=file use=@name / + +!-- Checkstyle XML Style Sheet by Stephane Bailliez sbaill...@apache.org -- +!-- Part of the Checkstyle distribution found at http://checkstyle.sourceforge.net -- +!-- Usage (generates
hadoop git commit: YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/branch-2 146abadb9 - 44aedad5d
YARN-3267. Timelineserver applies the ACL rules after applying the limit on the
number of records (Chang Li via jeagles)
(cherry picked from commit 8180e676abb2bb500a48b3a0c0809d2a807ab235)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/44aedad5
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/44aedad5
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/44aedad5
Branch: refs/heads/branch-2
Commit: 44aedad5ddc8069a6dba3eaf66ed54d612b21208
Parents: 146abad
Author: Jonathan Eagles jeag...@gmail.com
Authored: Fri Mar 13 12:04:30 2015 -0500
Committer: Jonathan Eagles jeag...@gmail.com
Committed: Fri Mar 13 12:05:21 2015 -0500
--
.../jobhistory/TestJobHistoryEventHandler.java | 14 +++---
.../mapred/TestMRTimelineEventHandling.java | 12 ++---
hadoop-yarn-project/CHANGES.txt | 3 ++
.../distributedshell/TestDistributedShell.java | 4 +-
.../server/timeline/LeveldbTimelineStore.java | 18 +--
.../server/timeline/MemoryTimelineStore.java| 12 -
.../server/timeline/TimelineDataManager.java| 50 +++-
.../yarn/server/timeline/TimelineReader.java| 3 +-
.../timeline/TestLeveldbTimelineStore.java | 16 +++
.../timeline/TestTimelineDataManager.java | 26 +-
.../server/timeline/TimelineStoreTestUtils.java | 33 +
11 files changed, 126 insertions(+), 65 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/44aedad5/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
--
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
index de35d84..43e3dbe 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
@@ -464,7 +464,7 @@ public class TestJobHistoryEventHandler {
t.appAttemptId, 200, t.containerId, nmhost, 3000, 4000),
currentTime - 10));
TimelineEntities entities = ts.getEntities(MAPREDUCE_JOB, null, null,
- null, null, null, null, null, null);
+ null, null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
TimelineEntity tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -480,7 +480,7 @@ public class TestJobHistoryEventHandler {
new HashMapJobACL, AccessControlList(), default),
currentTime + 10));
entities = ts.getEntities(MAPREDUCE_JOB, null, null, null,
- null, null, null, null, null);
+ null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -498,7 +498,7 @@ public class TestJobHistoryEventHandler {
new JobQueueChangeEvent(TypeConverter.fromYarn(t.jobId), q2),
currentTime - 20));
entities = ts.getEntities(MAPREDUCE_JOB, null, null, null,
- null, null, null, null, null);
+ null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -520,7 +520,7 @@ public class TestJobHistoryEventHandler {
new JobFinishedEvent(TypeConverter.fromYarn(t.jobId), 0, 0, 0, 0,
0, new Counters(), new Counters(), new Counters()),
currentTime));
entities = ts.getEntities(MAPREDUCE_JOB, null, null, null,
- null, null, null, null, null);
+ null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -546,7 +546,7 @@ public class TestJobHistoryEventHandler {
new JobUnsuccessfulCompletionEvent(TypeConverter.fromYarn(t.jobId),
0
hadoop git commit: YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles)
Repository: hadoop
Updated Branches:
refs/heads/trunk 387f271c8 - 8180e676a
YARN-3267. Timelineserver applies the ACL rules after applying the limit on the
number of records (Chang Li via jeagles)
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8180e676
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8180e676
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8180e676
Branch: refs/heads/trunk
Commit: 8180e676abb2bb500a48b3a0c0809d2a807ab235
Parents: 387f271
Author: Jonathan Eagles jeag...@gmail.com
Authored: Fri Mar 13 12:04:30 2015 -0500
Committer: Jonathan Eagles jeag...@gmail.com
Committed: Fri Mar 13 12:04:30 2015 -0500
--
.../jobhistory/TestJobHistoryEventHandler.java | 14 +++---
.../mapred/TestMRTimelineEventHandling.java | 12 ++---
hadoop-yarn-project/CHANGES.txt | 3 ++
.../distributedshell/TestDistributedShell.java | 4 +-
.../server/timeline/LeveldbTimelineStore.java | 18 +--
.../server/timeline/MemoryTimelineStore.java| 12 -
.../server/timeline/TimelineDataManager.java| 50 +++-
.../yarn/server/timeline/TimelineReader.java| 3 +-
.../timeline/TestLeveldbTimelineStore.java | 16 +++
.../timeline/TestTimelineDataManager.java | 26 +-
.../server/timeline/TimelineStoreTestUtils.java | 33 +
11 files changed, 126 insertions(+), 65 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/hadoop/blob/8180e676/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
--
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
index de35d84..43e3dbe 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java
@@ -464,7 +464,7 @@ public class TestJobHistoryEventHandler {
t.appAttemptId, 200, t.containerId, nmhost, 3000, 4000),
currentTime - 10));
TimelineEntities entities = ts.getEntities(MAPREDUCE_JOB, null, null,
- null, null, null, null, null, null);
+ null, null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
TimelineEntity tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -480,7 +480,7 @@ public class TestJobHistoryEventHandler {
new HashMapJobACL, AccessControlList(), default),
currentTime + 10));
entities = ts.getEntities(MAPREDUCE_JOB, null, null, null,
- null, null, null, null, null);
+ null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -498,7 +498,7 @@ public class TestJobHistoryEventHandler {
new JobQueueChangeEvent(TypeConverter.fromYarn(t.jobId), q2),
currentTime - 20));
entities = ts.getEntities(MAPREDUCE_JOB, null, null, null,
- null, null, null, null, null);
+ null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -520,7 +520,7 @@ public class TestJobHistoryEventHandler {
new JobFinishedEvent(TypeConverter.fromYarn(t.jobId), 0, 0, 0, 0,
0, new Counters(), new Counters(), new Counters()),
currentTime));
entities = ts.getEntities(MAPREDUCE_JOB, null, null, null,
- null, null, null, null, null);
+ null, null, null, null, null, null);
Assert.assertEquals(1, entities.getEntities().size());
tEntity = entities.getEntities().get(0);
Assert.assertEquals(t.jobId.toString(), tEntity.getEntityId());
@@ -546,7 +546,7 @@ public class TestJobHistoryEventHandler {
new JobUnsuccessfulCompletionEvent(TypeConverter.fromYarn(t.jobId),
0, 0, 0, JobStateInternal.KILLED.toString()), currentTime + 20
