[
https://issues.apache.org/jira/browse/HADOOP-17324?focusedWorklogId=509148&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-509148
]
ASF GitHub Bot logged work on HADOOP-17324:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 09/Nov/20 13:46
Start Date: 09/Nov/20 13:46
Worklog Time Spent: 10m
Work Description: steveloughran commented on a change in pull request
#2411:
URL: https://github.com/apache/hadoop/pull/2411#discussion_r519821934
##########
File path:
hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -44,6 +44,10 @@
import org.apache.hadoop.hdfs.web.WebHdfsTestUtil;
import org.apache.hadoop.hdfs.web.WebHdfsConstants;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
+
+import static org.junit.Assert.assertTrue;
Review comment:
unused, according to checkstyle
##########
File path:
hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -44,6 +44,10 @@
import org.apache.hadoop.hdfs.web.WebHdfsTestUtil;
import org.apache.hadoop.hdfs.web.WebHdfsConstants;
+import org.apache.hadoop.yarn.server.MiniYARNCluster;
Review comment:
should go straight after line 45 - no gap
##########
File path:
hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
Review comment:
we leave out all of test/resources to stop log4j files, site configs etc
getting onto the classpath of apps downstream -so making it impossible for them
to choose their own options
##########
File path:
hadoop-client-modules/hadoop-client-integration-tests/src/test/java/org/apache/hadoop/example/ITUseMiniCluster.java
##########
@@ -73,13 +78,22 @@ public void clusterUp() throws IOException {
.numDataNodes(3)
.build();
cluster.waitActive();
+
+ conf.set("yarn.scheduler.capacity.root.queues", "default");
+ conf.setInt("yarn.scheduler.capacity.root.default.capacity", 100);
+ yarnCluster = new MiniYARNCluster(getClass().getName(), 1, 1, 1, 1);
+ yarnCluster.init(conf);
+ yarnCluster.start();
}
@After
public void clusterDown() {
if (cluster != null) {
cluster.close();
}
+ if (yarnCluster != null) {
Review comment:
all services are closeable, so use `IOUtil.cleanupWithLogger()` & let it
handle null checks and exceptions
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 509148)
Time Spent: 2h 10m (was: 2h)
> Don't relocate org.bouncycastle in shaded client jars
> -----------------------------------------------------
>
> Key: HADOOP-17324
> URL: https://issues.apache.org/jira/browse/HADOOP-17324
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.3.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> When downstream apps depend on {{hadoop-client-api}},
> {{hadoop-client-runtime}} and {{hadoop-client-minicluster}}, it seems the
> {{MiniYARNCluster}} could have issue because
> {{org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException}}
> is not in any of the above jars.
> {code}
> Error: Caused by: sbt.ForkMain$ForkError: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.org.bouncycastle.operator.OperatorCreationException
> Error: at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> Error: at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
> Error: at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> Error: at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
> Error: at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:862)
> Error: at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Error: at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1296)
> Error: at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:339)
> Error: at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Error: at
> org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:353)
> Error: at
> org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:127)
> Error: at
> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:488)
> Error: at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Error: at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
> Error: at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:321)
> Error: at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Error: at
> org.apache.spark.deploy.yarn.BaseYarnClusterSuite.beforeAll(BaseYarnClusterSuite.scala:94)
> Error: at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> Error: at
> org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> Error: at
> org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> Error: at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:61)
> Error: at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:318)
> Error: at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:513)
> Error: at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:413)
> Error: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Error: at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Error: at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Error: at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
