On Mon, 2005-11-07 at 17:26 -0800, Paul Baclace wrote: > Rod Taylor wrote: > > The attached patches for Generator.java and Injector.java allow a > > specific temporary directory to be specified. This gives Nutch the full > > path to these temporary directories and seems to fix the "No input > > directories" issue when using a local filesystem with multiple task > > trackers. > > Is your patch with the new property mapred.temp.dir is meant to help > finding files that should not be separate between different > processes on the same host? Is the user id different?
Generate and Inject both issue 2 jobs. In order for the second job to find the files, the first job needs to write them in a predictable and common location. The current path doesn't seem to be enough even if all daemons are started within it. I believe it needs to be a common path for all hosts like mapred.system.dir which I considered using instead. NDFS accomplishes the above path finding by auto-prefixing any path not beginning with / with a /user/$USER. I didn't think it was appropriate for LocalFileSystem.java to be mucking around trying to automatically adjust paths to what the user may have intended. -- Rod Taylor <[EMAIL PROTECTED]>
