Hi
In our scenario there are around 30 machines from which we want to put
data into HDFS.
Now the approach we thought of initially was:
1. First tier : Agent which collect data from source then pass it to
avro sink.
2. Second tier: Lets call those agents 'collectors' which collect data
from First tier agents and then dump it to HDFS.
(Second tier agents are fewer in number say 4:1)
Instead of above topology if I simply use HDFS sink in first tier
agents. It can serve the purpose.
And also number of nodes are lesser (say 30) that won't hurt HDFS
namenode too much compared
to if number of nodes were say 1000.
But apart from that I don't say any advantage of adding the 2nd tier.
Is there any advantage I am missing in terms of failover, HDFS
performance or any other parameter?
Regards,
Jagadish
