On Jan 9, 2012, at 2:28 AM, Praveen Ramachandra wrote: > Hi, > > We were trying to design a multi-tenanted system using flume-ng, where each > logically independent data set is modelled through a channel going through > the system of collectors, aggregators and delivery agents (to end > destination). Each channel will carry data that logically belong together. > The requirement is that we should be able to bring up and tear down a channel > with ease. > > > When we completed the exercise, it turned out that we have to run a separate > Source/Sink, at a designated host/port combination for each channel. The > issue with this is that, it is an operational overhead that we have work with > net-ops to punch holes in the firewall to let tcp traffic flow on > non-standard ports. I would imagine that it would be the case in many > organizations as well. > > Two questions. > > 1. Did I get the modeling right with flume-ng > 2. Is there a better way to do it at a platform level > 2.1 I know if I can write a bunch of custom sinks/sources and > embed a notion of channel to which each events belong to in the message, I > can effectively mux and demux the events at either ends. > 2.2 Which means the default support for channel is also not of > much use
What is your target destination(s) for the tenants? Can they all flow through a single channel in Flume and then be delivered to the correct destination by a smarter sink at the end? Ralph
