[ http://issues.apache.org/jira/browse/HADOOP-123?page=comments#action_12415047 ]
alan wootton commented on HADOOP-123: ------------------------------------- I put my changes into the latest trunk, and built a patch. It doesn't work yet, but the outline is there. Notes: There is an added conf to override the hostname for nodes (DataNode and TaskTracker) There is a DNS hack so that java resolves all names in the form localhostN to 127.0.0.1 (MyNameService) This requires the system property 'sun.net.spi.nameservice.provider.1' to be set in the command line or in the ant script. dfs sim = RunDfsSim.java mapred sim = RunMapRedSim.java The dfs sim works pretty good. The MapRed sim is broken. It's always config issues. Generally you want for the TaskTrackers to use their local config's and avoid loading the default configs (which are all for localmode). I changed the build xml so that the 'test-cluster' target runs all tests that start with 'ClusterTest'. The ClusterTestDFS.java test is working but needs to be refactored to use RunDfsSim.java 'new JobConf(conf)' used to simply drop the conf on the floor, so there is a fix to Configuration for that. I had to add code to close the editlog in FSNamesystem or ClusterTestDFS.java fails on the second pass. Oops, the multiple editlogs patch, in FSDirectory, is also in this patch, but the default case changes nothing. There is a small change to the test-cluster target of build.xml to support the sims. Tell me what you think. > mini map/reduce cluster for junit tests > --------------------------------------- > > Key: HADOOP-123 > URL: http://issues.apache.org/jira/browse/HADOOP-123 > Project: Hadoop > Type: New Feature > Components: mapred > Versions: 0.3.0 > Reporter: Owen O'Malley > Assignee: Milind Bhandarkar > Fix For: 0.4.0 > Attachments: ClusterTestWordCount.patch > > I would like a single process map-reduce pseduo-distributed cluster that can > be used for unit tests. We should add a unit test with a word count > map/reduce. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
