[hadoop] branch branch-2.10 updated: HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein.
This is an automated email from the ASF dual-hosted git repository.
aajisaka pushed a commit to branch branch-2.10
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-2.10 by this push:
new 82b3caf HADOOP-17557. skip-dir option is not processed by Yetus.
Contributed by Ahmed Hussein.
82b3caf is described below
commit 82b3cafc1c519d6c74a3f3e7607c073dd907a6c6
Author: Akira Ajisaka
AuthorDate: Tue Mar 9 11:10:10 2021 +0900
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by
Ahmed Hussein.
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
---
dev-support/bin/test-patch | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-support/bin/test-patch b/dev-support/bin/test-patch
index 8ff8119..5faf472 100755
--- a/dev-support/bin/test-patch
+++ b/dev-support/bin/test-patch
@@ -15,4 +15,4 @@
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
-exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dir=dev-support "$@"
+exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dirs=dev-support "$@"
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.1 updated: HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein.
This is an automated email from the ASF dual-hosted git repository.
aajisaka pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new be253c2 HADOOP-17557. skip-dir option is not processed by Yetus.
Contributed by Ahmed Hussein.
be253c2 is described below
commit be253c2fd09b521cffe1a7f1250a0d3ccf8bb04b
Author: Akira Ajisaka
AuthorDate: Tue Mar 9 11:10:10 2021 +0900
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by
Ahmed Hussein.
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
---
dev-support/bin/test-patch | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-support/bin/test-patch b/dev-support/bin/test-patch
index 8ff8119..5faf472 100755
--- a/dev-support/bin/test-patch
+++ b/dev-support/bin/test-patch
@@ -15,4 +15,4 @@
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
-exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dir=dev-support "$@"
+exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dirs=dev-support "$@"
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein.
This is an automated email from the ASF dual-hosted git repository.
aajisaka pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 8d71ee1 HADOOP-17557. skip-dir option is not processed by Yetus.
Contributed by Ahmed Hussein.
8d71ee1 is described below
commit 8d71ee12d2511959a95edd51b937dbff4113e3d1
Author: Akira Ajisaka
AuthorDate: Tue Mar 9 11:10:10 2021 +0900
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by
Ahmed Hussein.
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
---
dev-support/bin/test-patch | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-support/bin/test-patch b/dev-support/bin/test-patch
index 8ff8119..5faf472 100755
--- a/dev-support/bin/test-patch
+++ b/dev-support/bin/test-patch
@@ -15,4 +15,4 @@
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
-exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dir=dev-support "$@"
+exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dirs=dev-support "$@"
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.3 updated: HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein.
This is an automated email from the ASF dual-hosted git repository.
aajisaka pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 2da6a96 HADOOP-17557. skip-dir option is not processed by Yetus.
Contributed by Ahmed Hussein.
2da6a96 is described below
commit 2da6a9682130683acc30820f49d0dc0831c699be
Author: Akira Ajisaka
AuthorDate: Tue Mar 9 11:10:10 2021 +0900
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by
Ahmed Hussein.
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
---
dev-support/bin/test-patch | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-support/bin/test-patch b/dev-support/bin/test-patch
index 8ff8119..5faf472 100755
--- a/dev-support/bin/test-patch
+++ b/dev-support/bin/test-patch
@@ -15,4 +15,4 @@
# limitations under the License.
BINDIR=$(cd -P -- "$(dirname -- "${BASH_SOURCE-0}")" >/dev/null && pwd -P)
-exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dir=dev-support "$@"
+exec "${BINDIR}/yetus-wrapper" test-patch --project=hadoop
--skip-dirs=dev-support "$@"
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch trunk updated (6a922f9 -> 63aa142)
This is an automated email from the ASF dual-hosted git repository. aajisaka pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/hadoop.git. from 6a922f9 HDFS-15873. Add namenode address in logs for block report (#2743) add 63aa142 HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. No new revisions were added by this update. Summary of changes: dev-support/bin/test-patch | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim Brennan.
This is an automated email from the ASF dual-hosted git repository.
ebadger pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new f30d109 YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV.
Contributed by Jim Brennan.
f30d109 is described below
commit f30d1092bc5e7275328b113a37e52f65785c9697
Author: Eric Badger
AuthorDate: Mon Mar 8 20:20:52 2021 +
YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by
Jim
Brennan.
---
.../containermanager/launcher/ContainerLaunch.java | 20 ++--
.../launcher/TestContainerLaunch.java | 112 +++--
2 files changed, 117 insertions(+), 15 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
index 84cc79e..15dddc0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
@@ -175,15 +175,13 @@ public class ContainerLaunch implements Callable
{
return var;
}
- private Map expandAllEnvironmentVars(
- ContainerLaunchContext launchContext, Path containerLogDir) {
-Map environment = launchContext.getEnvironment();
+ private void expandAllEnvironmentVars(
+ Map environment, Path containerLogDir) {
for (Entry entry : environment.entrySet()) {
String value = entry.getValue();
value = expandEnvironment(value, containerLogDir);
entry.setValue(value);
}
-return environment;
}
@Override
@@ -218,8 +216,10 @@ public class ContainerLaunch implements Callable {
}
launchContext.setCommands(newCmds);
- Map environment = expandAllEnvironmentVars(
- launchContext, containerLogDir);
+ // The actual expansion of environment variables happens after calling
+ // sanitizeEnv. This allows variables specified in NM_ADMIN_USER_ENV
+ // to reference user or container-defined variables.
+ Map environment = launchContext.getEnvironment();
// /// End of variable expansion
// Use this to track variables that are added to the environment by nm.
@@ -283,6 +283,8 @@ public class ContainerLaunch implements Callable {
containerLogDirs, localResources, nmPrivateClasspathJarDir,
nmEnvVars);
+expandAllEnvironmentVars(environment, containerLogDir);
+
prepareContainer(localResources, containerLocalDirs);
// Write out the environment
@@ -1564,13 +1566,13 @@ public class ContainerLaunch implements
Callable {
}
// variables here will be forced in, even if the container has
-// specified them.
+// specified them. Note: we do not track these in nmVars, to
+// allow them to be ordered properly if they reference variables
+// defined by the user.
String defEnvStr = conf.get(YarnConfiguration.DEFAULT_NM_ADMIN_USER_ENV);
Apps.setEnvFromInputProperty(environment,
YarnConfiguration.NM_ADMIN_USER_ENV, defEnvStr, conf,
File.pathSeparator);
-nmVars.addAll(Apps.getEnvVarsFromInputProperty(
-YarnConfiguration.NM_ADMIN_USER_ENV, defEnvStr, conf));
if (!Shell.WINDOWS) {
// maybe force path components
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
index d700850..0f856da 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
@@ -25,6 +25,7 @@ import static org.mockito.Matchers.any;
import static org.mockito.Mockito.*;
import java.io.BufferedReader;
+import java.io.DataOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.FileReader;
@@ -55,9 +56,8 @@ import
[hadoop] branch trunk updated: HDFS-15873. Add namenode address in logs for block report (#2743)
This is an automated email from the ASF dual-hosted git repository.
dineshc pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 6a922f9 HDFS-15873. Add namenode address in logs for block report
(#2743)
6a922f9 is described below
commit 6a922f98967ced0cbcf32d3acf6dfd10712e66a3
Author: litao
AuthorDate: Tue Mar 9 00:50:51 2021 +0800
HDFS-15873. Add namenode address in logs for block report (#2743)
---
.../org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
index 0cc9fac..3a54361 100755
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
@@ -436,7 +436,8 @@ class BPServiceActor implements Runnable {
final int nCmds = cmds.size();
LOG.info((success ? "S" : "Uns") +
"uccessfully sent block report 0x" +
- Long.toHexString(reportId) + ", containing " + reports.length +
+ Long.toHexString(reportId) + " to namenode: " + nnAddr +
+ ", containing " + reports.length +
" storage report(s), of which we sent " + numReportsSent + "." +
" The reports had " + totalBlockCount +
" total blocks and used " + numRPCs +
@@ -930,10 +931,12 @@ class BPServiceActor implements Runnable {
void triggerBlockReport(BlockReportOptions options) {
if (options.isIncremental()) {
- LOG.info(bpos.toString() + ": scheduling an incremental block report.");
+ LOG.info(bpos.toString() + ": scheduling an incremental block report " +
+ "to namenode: " + nnAddr + ".");
ibrManager.triggerIBR(true);
} else {
- LOG.info(bpos.toString() + ": scheduling a full block report.");
+ LOG.info(bpos.toString() + ": scheduling a full block report " +
+ "to namenode: " + nnAddr + ".");
synchronized(ibrManager) {
scheduler.forceFullBlockReportNow();
ibrManager.notifyAll();
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth.
This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 0b8bfc5 YARN-10672. All testcases in TestReservations are flaky.
Contributed by Szilard Nemeth.
0b8bfc5 is described below
commit 0b8bfc50c6e759a5693c409cb760b6ad4b316f11
Author: Peter Bacsko
AuthorDate: Mon Mar 8 14:32:13 2021 +0100
YARN-10672. All testcases in TestReservations are flaky. Contributed by
Szilard Nemeth.
---
.../server/resourcemanager/scheduler/capacity/TestReservations.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
index 625b1e7..53c2318 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
@@ -106,7 +106,6 @@ public class TestReservations {
CapacityScheduler spyCs = new CapacityScheduler();
cs = spy(spyCs);
rmContext = TestUtils.getMockRMContext();
-
}
private void setup(CapacitySchedulerConfiguration csConf) throws Exception {
@@ -115,6 +114,9 @@ public class TestReservations {
private void setup(CapacitySchedulerConfiguration csConf,
boolean addUserLimits) throws Exception {
+//All stub calls on the spy object of the 'cs' field should happen
+//before cs.start() is invoked. See YARN-10672 for more details.
+when(cs.getNumClusterNodes()).thenReturn(3);
csConf.setBoolean(CapacitySchedulerConfiguration.ENABLE_USER_METRICS,
true);
final String newRoot = "root" + System.currentTimeMillis();
@@ -157,8 +159,6 @@ public class TestReservations {
cs.setRMContext(spyRMContext);
cs.init(csConf);
cs.start();
-
-when(cs.getNumClusterNodes()).thenReturn(3);
}
private static final String A = "a";
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
[hadoop] branch branch-3.2 updated: YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu.
This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 0d032d7 YARN-10642. Race condition: AsyncDispatcher can get stuck by
the changes introduced in YARN-8995. Contributed by zhengchenyu.
0d032d7 is described below
commit 0d032d7b6856903d754a588ca7e46184f869d4c4
Author: Peter Bacsko
AuthorDate: Mon Mar 8 14:22:14 2021 +0100
YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes
introduced in YARN-8995. Contributed by zhengchenyu.
---
.../apache/hadoop/yarn/event/AsyncDispatcher.java | 19 ---
.../hadoop/yarn/event/TestAsyncDispatcher.java | 58 +-
2 files changed, 68 insertions(+), 9 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
index 81448c5..acfba27 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
@@ -20,11 +20,11 @@ package org.apache.hadoop.yarn.event;
import java.util.ArrayList;
import java.util.HashMap;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
-import java.util.stream.Collectors;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -265,11 +265,16 @@ public class AsyncDispatcher extends AbstractService
implements Dispatcher {
}
class GenericEventHandler implements EventHandler {
-private void printEventQueueDetails(BlockingQueue queue) {
- Map counterMap = eventQueue.stream().
- collect(Collectors.
- groupingBy(e -> e.getType(), Collectors.counting())
- );
+private void printEventQueueDetails() {
+ Iterator iterator = eventQueue.iterator();
+ Map counterMap = new HashMap<>();
+ while (iterator.hasNext()) {
+Enum eventType = iterator.next().getType();
+if (!counterMap.containsKey(eventType)) {
+ counterMap.put(eventType, 0L);
+}
+counterMap.put(eventType, counterMap.get(eventType) + 1);
+ }
for (Map.Entry entry : counterMap.entrySet()) {
long num = entry.getValue();
LOG.info("Event type: " + entry.getKey()
@@ -292,7 +297,7 @@ public class AsyncDispatcher extends AbstractService
implements Dispatcher {
if (qSize != 0 && qSize % detailsInterval == 0
&& lastEventDetailsQueueSizeLogged != qSize) {
lastEventDetailsQueueSizeLogged = qSize;
-printEventQueueDetails(eventQueue);
+printEventQueueDetails();
printTrigger = true;
}
int remCapacity = eventQueue.remainingCapacity();
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
index 53e2d05..88855fc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
@@ -97,12 +97,23 @@ public class TestAsyncDispatcher {
}
private static class TestHandler implements EventHandler {
+
+private long sleepTime = 1500;
+
+TestHandler() {
+}
+
+TestHandler(long sleepTime) {
+ this.sleepTime = sleepTime;
+}
+
@Override
public void handle(Event event) {
try {
// As long as 1 events queued
-Thread.sleep(1500);
- } catch (InterruptedException e) {}
+Thread.sleep(this.sleepTime);
+ } catch (InterruptedException e) {
+ }
}
}
@@ -170,11 +181,54 @@ public class TestAsyncDispatcher {
//Make sure more than one event to take
verify(log, atLeastOnce()).
info("Latest dispatch event type: TestEventType");
+} finally {
+ //... restore logger object
+ logger.set(null, oldLog);
dispatcher.stop();
+}
+ }
+
+ //Test print dispatcher details when the blocking queue is heavy
+ @Test(timeout = 6)
+ public void testPrintDispatcherEventDetailsAvoidDeadLoop() throws Exception {
+for (int i = 0; i < 5; i++) {
+ testPrintDispatcherEventDetailsAvoidDeadLoopInternal();
+}
+ }
+
+ public void testPrintDispatcherEventDetailsAvoidDeadLoopInternal()
+ throws Exception {
[hadoop] branch branch-3.3 updated: YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu.
This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new eb05d49 YARN-10642. Race condition: AsyncDispatcher can get stuck by
the changes introduced in YARN-8995. Contributed by zhengchenyu.
eb05d49 is described below
commit eb05d49349d223052db74f528b239a5392584934
Author: Peter Bacsko
AuthorDate: Mon Mar 8 12:36:08 2021 +0100
YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes
introduced in YARN-8995. Contributed by zhengchenyu.
---
.../apache/hadoop/yarn/event/AsyncDispatcher.java | 19 ---
.../hadoop/yarn/event/TestAsyncDispatcher.java | 58 +-
2 files changed, 68 insertions(+), 9 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
index 79ad464..f9deab0 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/AsyncDispatcher.java
@@ -20,11 +20,11 @@ package org.apache.hadoop.yarn.event;
import java.util.ArrayList;
import java.util.HashMap;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
-import java.util.stream.Collectors;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -268,11 +268,16 @@ public class AsyncDispatcher extends AbstractService
implements Dispatcher {
}
class GenericEventHandler implements EventHandler {
-private void printEventQueueDetails(BlockingQueue queue) {
- Map counterMap = eventQueue.stream().
- collect(Collectors.
- groupingBy(e -> e.getType(), Collectors.counting())
- );
+private void printEventQueueDetails() {
+ Iterator iterator = eventQueue.iterator();
+ Map counterMap = new HashMap<>();
+ while (iterator.hasNext()) {
+Enum eventType = iterator.next().getType();
+if (!counterMap.containsKey(eventType)) {
+ counterMap.put(eventType, 0L);
+}
+counterMap.put(eventType, counterMap.get(eventType) + 1);
+ }
for (Map.Entry entry : counterMap.entrySet()) {
long num = entry.getValue();
LOG.info("Event type: " + entry.getKey()
@@ -295,7 +300,7 @@ public class AsyncDispatcher extends AbstractService
implements Dispatcher {
if (qSize != 0 && qSize % detailsInterval == 0
&& lastEventDetailsQueueSizeLogged != qSize) {
lastEventDetailsQueueSizeLogged = qSize;
-printEventQueueDetails(eventQueue);
+printEventQueueDetails();
printTrigger = true;
}
int remCapacity = eventQueue.remainingCapacity();
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
index 762e228..55ddd12 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/event/TestAsyncDispatcher.java
@@ -97,12 +97,23 @@ public class TestAsyncDispatcher {
}
private static class TestHandler implements EventHandler {
+
+private long sleepTime = 1500;
+
+TestHandler() {
+}
+
+TestHandler(long sleepTime) {
+ this.sleepTime = sleepTime;
+}
+
@Override
public void handle(Event event) {
try {
// As long as 1 events queued
-Thread.sleep(1500);
- } catch (InterruptedException e) {}
+Thread.sleep(this.sleepTime);
+ } catch (InterruptedException e) {
+ }
}
}
@@ -170,11 +181,54 @@ public class TestAsyncDispatcher {
//Make sure more than one event to take
verify(log, atLeastOnce()).
info("Latest dispatch event type: TestEventType");
+} finally {
+ //... restore logger object
+ logger.set(null, oldLog);
dispatcher.stop();
+}
+ }
+
+ //Test print dispatcher details when the blocking queue is heavy
+ @Test(timeout = 6)
+ public void testPrintDispatcherEventDetailsAvoidDeadLoop() throws Exception {
+for (int i = 0; i < 5; i++) {
+ testPrintDispatcherEventDetailsAvoidDeadLoopInternal();
+}
+ }
+
+ public void testPrintDispatcherEventDetailsAvoidDeadLoopInternal()
+ throws Exception {
+YarnConfiguration conf
[hadoop] branch trunk updated: YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu.
This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/trunk by this push:
new 04cd311 YARN-10658. CapacityScheduler QueueInfo add queue path field
to avoid ambiguous QueueName. Contributed by Qi Zhu.
04cd311 is described below
commit 04cd3115ba3f078eea7e76c3bb7d5228266378d8
Author: Peter Bacsko
AuthorDate: Mon Mar 8 12:29:44 2021 +0100
YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid
ambiguous QueueName. Contributed by Qi Zhu.
---
.../apache/hadoop/yarn/api/records/QueueInfo.java | 26
.../src/main/proto/yarn_protos.proto | 1 +
.../apache/hadoop/yarn/client/cli/QueueCLI.java| 2 ++
.../hadoop/yarn/client/ProtocolHATestBase.java | 2 +-
.../apache/hadoop/yarn/client/cli/TestYarnCLI.java | 10 +--
.../yarn/api/records/impl/pb/QueueInfoPBImpl.java | 16 ++
.../apache/hadoop/yarn/api/TestPBImplRecords.java | 3 +-
.../scheduler/capacity/AbstractCSQueue.java| 1 +
.../TestConfigurationMutationACLPolicies.java | 17 ++-
.../scheduler/TestSchedulerApplicationAttempt.java | 3 +-
.../scheduler/capacity/TestCapacityScheduler.java | 6
.../TestCapacitySchedulerNewQueueAutoCreation.java | 35 ++
.../scheduler/capacity/TestLeafQueue.java | 15 ++
13 files changed, 114 insertions(+), 23 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
index 803adad..eb703cd 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueInfo.java
@@ -54,7 +54,8 @@ public abstract class QueueInfo {
@Private
@Unstable
- public static QueueInfo newInstance(String queueName, float capacity,
+ public static QueueInfo newInstance(String queueName,
+ String queuePath, float capacity,
float maximumCapacity, float currentCapacity,
List childQueues, List applications,
QueueState queueState, Set accessibleNodeLabels,
@@ -62,6 +63,7 @@ public abstract class QueueInfo {
boolean preemptionDisabled, float weight) {
QueueInfo queueInfo = Records.newRecord(QueueInfo.class);
queueInfo.setQueueName(queueName);
+queueInfo.setQueuePath(queuePath);
queueInfo.setCapacity(capacity);
queueInfo.setMaximumCapacity(maximumCapacity);
queueInfo.setCurrentCapacity(currentCapacity);
@@ -78,14 +80,15 @@ public abstract class QueueInfo {
@Private
@Unstable
- public static QueueInfo newInstance(String queueName, float capacity,
+ public static QueueInfo newInstance(String queueName,
+ String queuePath, float capacity,
float maximumCapacity, float currentCapacity,
List childQueues, List applications,
QueueState queueState, Set accessibleNodeLabels,
String defaultNodeLabelExpression, QueueStatistics queueStatistics,
boolean preemptionDisabled, float weight,
Map queueConfigurations) {
-QueueInfo queueInfo = QueueInfo.newInstance(queueName, capacity,
+QueueInfo queueInfo = QueueInfo.newInstance(queueName, queuePath, capacity,
maximumCapacity, currentCapacity,
childQueues, applications,
queueState, accessibleNodeLabels,
@@ -97,7 +100,8 @@ public abstract class QueueInfo {
@Private
@Unstable
- public static QueueInfo newInstance(String queueName, float capacity,
+ public static QueueInfo newInstance(String queueName,
+ String queuePath, float capacity,
float maximumCapacity, float currentCapacity,
List childQueues, List applications,
QueueState queueState, Set accessibleNodeLabels,
@@ -105,7 +109,7 @@ public abstract class QueueInfo {
boolean preemptionDisabled, float weight,
Map queueConfigurations,
boolean intraQueuePreemptionDisabled) {
-QueueInfo queueInfo = QueueInfo.newInstance(queueName, capacity,
+QueueInfo queueInfo = QueueInfo.newInstance(queueName, queuePath, capacity,
maximumCapacity, currentCapacity,
childQueues, applications,
queueState, accessibleNodeLabels,
@@ -126,6 +130,18 @@ public abstract class QueueInfo {
@Private
@Unstable
public abstract void setQueueName(String queueName);
+
+ /**
+ * Get the path of the queue.
+ * @return path of the queue
+ */
+ @Public
+ @Stable
+ public abstract String getQueuePath();
+
+ @Private
+ @Unstable
+ public abstract void setQueuePath(String queuePath);
/**
* Get the configured capacity of the queue.
diff --git
[hadoop] branch branch-3.3 updated: YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth.
This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 066f89a YARN-10672. All testcases in TestReservations are flaky.
Contributed by Szilard Nemeth.
066f89a is described below
commit 066f89af019811bc3b16703f89125a10d2b2a07e
Author: Peter Bacsko
AuthorDate: Mon Mar 8 11:42:59 2021 +0100
YARN-10672. All testcases in TestReservations are flaky. Contributed by
Szilard Nemeth.
---
.../server/resourcemanager/scheduler/capacity/TestReservations.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
index a9cee2d..1168f64 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
@@ -105,7 +105,6 @@ public class TestReservations {
CapacityScheduler spyCs = new CapacityScheduler();
cs = spy(spyCs);
rmContext = TestUtils.getMockRMContext();
-
}
private void setup(CapacitySchedulerConfiguration csConf) throws Exception {
@@ -114,6 +113,9 @@ public class TestReservations {
private void setup(CapacitySchedulerConfiguration csConf,
boolean addUserLimits) throws Exception {
+//All stub calls on the spy object of the 'cs' field should happen
+//before cs.start() is invoked. See YARN-10672 for more details.
+when(cs.getNumClusterNodes()).thenReturn(3);
csConf.setBoolean(CapacitySchedulerConfiguration.ENABLE_USER_METRICS,
true);
final String newRoot = "root" + System.currentTimeMillis();
@@ -156,8 +158,6 @@ public class TestReservations {
cs.setRMContext(spyRMContext);
cs.init(csConf);
cs.start();
-
-when(cs.getNumClusterNodes()).thenReturn(3);
}
private static final String A = "a";
-
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org
