The attached screenshot will shows how flume will work, and also you can consider RabbitMQ, as it persistent too..
∞ Shashwat Shriparv On Fri, Jan 11, 2013 at 10:24 AM, Mohit Anchlia <[email protected]>wrote: > Have you looked at flume? > > Sent from my iPhone > > On Jan 10, 2013, at 7:12 PM, Panshul Whisper <[email protected]> > wrote: > > > Hello, > > > > I have a hadoop cluster setup of 10 nodes and I an in need of > implementing queues in the cluster for receiving high volumes of data. > > Please suggest what will be more efficient to use in the case of > receiving 24 Million Json files.. approx 5 KB each in every 24 hours : > > 1. Using Capacity Scheduler > > 2. Implementing RabbitMQ and receive data from them using Spring > Integration Data pipe lines. > > > > I cannot afford to loose any of the JSON files received. > > > > Thanking You, > > > > -- > > Regards, > > Ouch Whisper > > 010101010101 >
<<attachment: 2013-01-11_1031.png>>
