Hey everyone, i deal with hadoop since a few weeks to build up a cluster with hdfs. I was looking for several Monitoring tools to observe my cluster and find a good solution with ganglia+nagios. To complete the monitoring part of the cluster, i am looking for an Log collection tool, which store the log files of the nodes centralized. I have tested Chukwa and Facebook's Scribe, but both are not that type of simple storing log files, in my opinion they are too big, only for such a job.
So i've thinking about writing an own LogCollector. I didn't want something special. My idea is, to build a deamon, which could be installed on every node in the cluster and onxml-file, which describes which log files have to be collected. The daemon should collect, in configured time interval, all needed log files and store them using the Java API in HDFS. This was just an idea for a simple LogCollector and it would cool if you can give me some opinion about this or whether such a LogCollector exits. Kind regards, Patrick
