[
https://issues.apache.org/jira/browse/HADOOP-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660681#action_12660681
]
Hemanth Yamijala commented on HADOOP-4830:
------------------------------------------
Looking good. A few comments:
- We are iterating over the task list to get the number of running tasks in
ControlledMapReduceJob.getRunningTasksCount(). We check if the task is running
using TaskInProgress.isRunning(). This method of computation seems like it
would not be different from JobInProgress.runningMaps() or
JobInProgress.runningReduces(). Can you please check if there is a difference ?
- There are a few TODOs in the comments in ControlledMapReduceJob. These should
be removed, or alternatively we should add a more descriptive comment about a
possible improvement.
- In writeFile, some lines related to replication are commented. These can be
removed.
- The RunningJob variable, rJob, can be private and not package private
When I ran the test on my system, it failed because of missing some classes
related to Jetty:
{code}
org/mortbay/jetty/servlet/Context
java.lang.NoClassDefFoundError: org/mortbay/jetty/servlet/Context
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:220)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:202)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:955)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:275)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:119)
at
org.apache.hadoop.mapred.ClusterWithCapacityScheduler.startCluster(ClusterWithCapacityScheduler.java:101)
at
org.apache.hadoop.mapred.TestQueueCapacities.testSingleQueue(TestQueueCapacities.java:54)
{code}
This may be that we need to fix something in ivy.xml of the capacity scheduler.
Copying the ivy.xml from streaming ran the test successfully.
> Have end to end tests based on MiniMRCluster to verify that queue capacities
> are honoured.
> ------------------------------------------------------------------------------------------
>
> Key: HADOOP-4830
> URL: https://issues.apache.org/jira/browse/HADOOP-4830
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Reporter: Vinod K V
> Assignee: Vinod K V
> Attachments: HADOOP-4830-20081222-svn.2, HADOOP-4830-20081229-svn.txt
>
>
> At present, we only have unit tests that make use of FakeTaskManager and that
> only test the proper functionality of capacity scheduler in isolation. Many
> issues unearthed recently proved that this is not enough and that it is
> required to have end-to-end tests so that real JT is brought into the picture
> and with that the interaction of the scheduler with JT. This issue along with
> few other related jiras should automate and replace the end-to-end tests that
> are now manually done by QA, using MiniMRCluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
