Re: Small patch to improve mapred.pipes

[Arun C Murthy]

Leon,

On Aug 9, 2008, at 12:08 PM, Leon Mergen wrote:

Hello,

Working with the .mapred.pipes class a bit today, I found one bug and one

possible interface improvement:

Do you mind opening a jira and attaching this patch there?

thanks,
Arun

- Application.java in its constructor assumes that
DistributedCache.getLocalCacheFiles

() always returns non-zero, and returns an array with at least one element

-- appropriate checks and exceptions should be thrown here.

- Submitter.java provides a "submitJob ()" method, yet it acts like it's a runJob method (and also invokes mapred.JobClient.runClient ()) -- it should provide two interface methods, one runJob () and one submitJob (), who act

just like the JobClient counterparts.

Here is the small patch that implements both changes, based on the 0.17.1 release source -- just in case anyone cares for this minor improvement.

diff -r ./src/java/org/apache/hadoop/mapred/pipes/Application.java

/usr/local/hadoop/src/java/org/apache/hadoop/mapred/pipes/ Application.java

33a34

import org.apache.hadoop.fs.Path;

79c80,86
<     String executable =
DistributedCache.getLocalCacheFiles(conf)[0].toString();
---

Path[] executablePath = DistributedCache.getLocalCacheFiles (conf);

   if (executablePath == null || executablePath.length == 0) {
           throw new IOException ("Unable to retrieve path to

executable");

   }

   String executable = executablePath[0].toString();

diff -r ./src/java/org/apache/hadoop/mapred/pipes/Submitter.java

/usr/local/hadoop/src/java/org/apache/hadoop/mapred/pipes/ Submitter.java

213c213

< public static RunningJob submitJob(JobConf conf) throws IOException {

---

  private static void prepareJob(JobConf conf) throws IOException {

215a216

227a229

232a235

244a248

252a257

261d265
<     return JobClient.runJob(conf);
263a268,278

public static RunningJob submitJob (JobConf conf, JobClient jc) throws

IOException {

   prepareJob (conf);
   return jc.submitJob (conf);
 }

 public static RunningJob runJob (JobConf conf) throws IOException {
   prepareJob (conf);
   return JobClient.runJob (conf);
 }

322a338

357a374

417c434
<       submitJob(conf);
---

     runJob(conf);

diff -r ./src/test/org/apache/hadoop/mapred/pipes/TestPipes.java

/usr/local/hadoop/src/test/org/apache/hadoop/mapred/pipes/ TestPipes.java

152c152
<       RunningJob result = Submitter.submitJob(job);
---

     RunningJob result = Submitter.runJob(job);

--
Leon Mergen
http://www.solatis.com