On Mon, Jul 20, 2009 at 5:02 PM, Aaron Kimball <[email protected]> wrote:
> There's likely another gotcha regarding the fact that various logs and job > config files are written to the _logs directory under the output directory. That can be turned off by setting hadoop.job.history.user.location to none. > This might be a bit of a frustrating endeavour since you're trying to > override behaviour that's been baked in to Hadoop for a while. It isn't baked into Hadoop, but rather the FileOutputFormats. If you write your own OutputFormat, it is easy to avoid this restriction. However, the restriction is there to help you avoid mistakes, so if you don't do the checks, you can do bad things to your data. *smile* > Why in > particular do you need all your jobs to emit to a common directory? You > could probably save yourself some headache by writing to subdirectories of > a > common dir. That sounds very reasonable. -- Owen
