You can use scribe or flume to collect log data and integrated with hadoop.
2012/8/4 Nguyen Manh Tien <tien.nguyenm...@gmail.com> > Hi, > I plan to streaming logs data HDFS using many writer, each writer write a > stream of data to a HDFS file (may rotate) > > I wonder how many concurrent writer i should use? > And if you have that experience please share to me : hadoop cluster size, > number of writer, replication. > > Thanks. > Tien >
