There is a single log4j file on a publicly accessible NFS path. I haven't been able to try restarting to task trackers because the cluster has to stay up all the time.
Setting the logging level programmatically works well though. I can turn it on and off with a Hadoop configuration parameter, which lets me set the logging level on a per job basis instead of setting one level per class across the entire cluster. On Wed, Dec 15, 2010 at 5:52 PM, Aaron Kimball <[email protected]> wrote: > How is the central log4j file made available to the tasks? After you make > your changes to the configuration file, does it help if you restart the > task > trackers? > > You could also try setting the log level programmatically in your "void > setup(Context)" method: > > @Override > protected void setup(Context context) { > logger.setLevel(Level.DEBUG); > } > > - Aaron > > On Wed, Dec 15, 2010 at 2:23 PM, W.P. McNeill <[email protected]> wrote: > > > I'm running on a cluster. I'm trying to write to the log files on the > > cluster machines, the ones that are visible through the jobtracker web > > interface. > > > > The log4j file I gave excerpts from is a central one for the cluster. > > > > On Wed, Dec 15, 2010 at 1:38 PM, Aaron Kimball <[email protected]> > > wrote: > > > > > W. P., > > > > > > How are you running your Reducer? Is everything running in standalone > > mode > > > (all mappers/reducers in the same process as the launching > application)? > > Or > > > are you running this in pseudo-distributed mode or on a remote cluster? > > > > > > Depending on the application's configuration, log4j configuration could > > be > > > read from one of many different places. > > > > > > Furthermore, where are you expecting your output? If you're running in > > > pseudo-distributed (or fully distributed) mode, mapper / reducer tasks > > will > > > not emit output back to the console of the launching application. That > > > only > > > happens in local mode. In the distributed flavors, you'll see a > different > > > file for each task attempt containing its log output, on the machine > > where > > > the task executed. These files can be accessed through the web UI at > > > http://jobtracker:50030/ -- click on the job, then the task, then the > > task > > > attempt, then "syslog" in the right-most column. > > > > > > - Aaron > > > > > > On Mon, Dec 13, 2010 at 10:05 AM, W.P. McNeill <[email protected]> > > wrote: > > > > > > > I would like to use Hadoop's Log4j infrastructure to do logging from > my > > > > map/reduce application. I think I've got everything set up > correctly, > > > but > > > > I > > > > am still unable to specify the logging level I want. > > > > > > > > By default Hadoop is set up to log at level INFO. The first line of > > its > > > > log4j.properties file looks like this: > > > > > > > > hadoop.root.logger=INFO,console > > > > > > > > > > > > I have an application whose reducer looks like this: > > > > > > > > package com.me; > > > > > > > > public class MyReducer<...> extends Reducer<...> { > > > > private static Logger logger = > > > > Logger.getLogger(MyReducer.class.getName()); > > > > > > > > ... > > > > protected void reduce(...) { > > > > logger.debug("My message"); > > > > ... > > > > } > > > > } > > > > > > > > > > > > I've added the following line to the Hadoop log4j.properties file: > > > > > > > > log4j.logger.com.me.MyReducer=DEBUG > > > > > > > > > > > > I expect the Hadoop system to log at level INFO, but my application > to > > > log > > > > at level DEBUG, so that I see "My message" in the logs for the > reducer > > > > task. > > > > However, my application does not produce any log4j output. If I > > change > > > > the > > > > line in my reducer to read logger.info("My message") the message > does > > > get > > > > logged, so somehow I'm failing to specify that log level for this > > class. > > > > > > > > I've also tried changing the log4j line for my app to > > > > read log4j.logger.com.me.MyReducer=DEBUG,console and get the same > > result. > > > > > > > > I've been through the Hadoop and log4j documentation and I can't > figure > > > out > > > > what I'm doing wrong. Any suggestions? > > > > > > > > Thanks. > > > > > > > > > >
