Hi Kim, How are you handling realtime collection? I am okay with a few minutes delay, but the collector is collecting only once every hour. What configuration I should use to set the collector to collect every minute?
Have you set up the flume yet? How are you handling agent dying? Thanks, Chalcy ________________________________________ From: Kim, Jongkook [jongkook....@citi.com] Sent: Tuesday, February 14, 2012 2:25 PM To: 'flume-user@incubator.apache.org' Subject: Scale of a flume collector Hi all. I'm in the middle of hardware provisioning for flume-hbase-hadoop solution. The plan is that flume agents collect and pass log data to collectors and the collectors write data into hbase using sink. The question is a flume collector's scale. Flume agents:250 Data receiving ratio: 5.78MB/second Data writing ratio: 17.9MB/second Number of data nodes: 12 This system will be used to provide real-time use case, so there shouldn't be delay. How many collectors required to handle this request? Thanks in advance,
