The first part looks fine. I dug in, and see that the transpose() method ultimately does not use the configuration that was configured and makes its own. That is the underlying issue. Maybe Jake can comment more.
On Tue, Jun 15, 2010 at 10:28 AM, Kris Jack <[email protected]> wrote: > Hi Sean, > > I'm calling getConf() and using it to configure my DistributedRowMatrix. > > // > Configuration originalConf = getConf(); > String inputPathString = originalConf.get("mapred.input.dir"); > String outputTmpPathString = parsedArgs.get("--tempDir"); > int numDocs = Integer.parseInt(parsedArgs.get("--numDocs")); > int numTerms = Integer.parseInt(parsedArgs.get("--numTerms")); > > DistributedRowMatrix text = new DistributedRowMatrix(new > Path(inputPathString), new Path(outputTmpPathString), numDocs, numTerms); > > text.configure(new JobConf(getConf())); > > DistributedRowMatrix transpose = text.transpose(); > // > > On debugging, I notice that originalConf object does not have the values > that I sent in through the command line. When text.transpose() is called, > the transpose job's conf doesn't have the right values for the mappers and > reducers neither. Where am I supposed to get the command line values to be > used by these jobs? > > Thanks, > Kris >
