What we have done is to configure syslog to forward all logging to a log server. All logs are sent using UDP. Works like a charm, we have a directory per node and for some logs, we have all mixed in a file... Check out syslog documentation...
On Fri, Apr 23, 2010 at 2:38 PM, Patrick Datko <[email protected]> wrote: > Hey everyone, > > i deal with hadoop since a few weeks to build up a cluster with hdfs. I > was looking for several Monitoring tools to observe my cluster and find > a good solution with ganglia+nagios. To complete the monitoring part of > the cluster, i am looking for an Log collection tool, which store the > log files of the nodes centralized. I have tested Chukwa and Facebook's > Scribe, but both are not that type of simple storing log files, in my > opinion they are too big, only for such a job. > > So i've thinking about writing an own LogCollector. I didn't want > something special. My idea is, to build a deamon, which could be > installed on every node in the cluster and onxml-file, which describes > which log files have to be collected. The daemon should collect, in > configured time interval, all needed log files and store them using the > Java API in HDFS. > > This was just an idea for a simple LogCollector and it would cool if you > can give me some opinion about this or whether such a LogCollector > exits. > > Kind regards, > Patrick > > -- http://www.neko-consulting.com Ego sum quis ego servo "Je suis ce que je protège" "I am what I protect"
