[ http://issues.apache.org/jira/browse/HADOOP-123?page=all ]
Milind Bhandarkar updated HADOOP-123:
-------------------------------------
Attachment: minimr.patch
I have added a patch that now includes a mini-map-reduce cluster in unit tests.
There are three unit tests added that use the mini-map-reduce cluster. Main
changes (other than test additions) are: Now jobtracker, and tasktrackers can
be cleanly shut down (earlier ctrl-c was the only way). Class path changes in
test.classpath (job-tracker and task-tracker need webapps, and a temporary
overriding mechanism to overriding src/test/hadoop-site.xml). Multiple
task-trackers can be spawned on the same node with their respective
mapred.local.dir, therefore unless explicitly overridden in hadoop-site.xml,
the task uses task-tracker's mapred.local.dir.
The three new unit tests take 15 seconds, 115 seconds, and 119 seconds
respectively, on my linux box. They test mini MR cluster's bringup-shutdown, a
map-reduce job with local file system, and a map-reduce job with mini-DFS
cluster.
By the way, Hadoop now satisfies an important requirement of being a parallel
programming infrastructure. I have included a map-reduce job that estimates the
value of PI using monte-carlo method :-)
> mini map/reduce cluster for junit tests
> ---------------------------------------
>
> Key: HADOOP-123
> URL: http://issues.apache.org/jira/browse/HADOOP-123
> Project: Hadoop
> Type: New Feature
> Components: mapred
> Versions: 0.3.0
> Reporter: Owen O'Malley
> Assignee: Milind Bhandarkar
> Fix For: 0.4.0
> Attachments: ClusterTestWordCount.patch, minimr.patch
>
> I would like a single process map-reduce pseduo-distributed cluster that can
> be used for unit tests. We should add a unit test with a word count
> map/reduce.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
