Great thanks this clears everything thanks! On Wed, Oct 10, 2012 at 4:45 PM, Harish Mandala <[email protected]>wrote:
> You would use something like puppet or chef to push such config files > around. > > Regards, > Harish > > > On Wed, Oct 10, 2012 at 2:19 PM, Camp, Roy <[email protected]> wrote: > >> You have to manually start each node with its specific configuration. >> However, you can put the configuration for all your different setups into >> one config file, but you will still need to place a copy of it on every >> machine. Simply define which agent config to use with the --name param >> when starting. **** >> >> ** ** >> >> Thanks,**** >> >> ** ** >> >> Roy**** >> >> ** ** >> >> ** ** >> >> *From:* Juan Gentile [mailto:[email protected]] >> *Sent:* Wednesday, October 10, 2012 9:54 AM >> *To:* [email protected] >> *Subject:* Re: Flume-ng - Distributed**** >> >> ** ** >> >> Thank you both very much, I've been reading the documentation you sent me >> and this brings another question, is there a way to submit my flume >> configuration to a cluster or I have to manually start up each node with >> the its specific configuration?**** >> >> ** ** >> >> Thank you!**** >> >> On Wed, Oct 10, 2012 at 1:51 AM, Mike Percy <[email protected]> wrote:*** >> * >> >> +1 on what Roy said, with a minor terminology quibble: in Flume NG the >> Avro collector component is called the Avro Source.**** >> >> ** ** >> >> Also, here are links to the docs with working image links and table of >> contents:**** >> >> ** ** >> >> http://flume.apache.org/FlumeUserGuide.html**** >> >> http://flume.apache.org/FlumeDeveloperGuide.html**** >> >> ** ** >> >> Regards,**** >> >> Mike**** >> >> ** ** >> >> On Tue, Oct 9, 2012 at 5:52 PM, Camp, Roy <[email protected]> wrote:**** >> >> You would run a flume-ng instance on each node with an avro-sink. Then >> on your collector machine you will run another flume-ng instance with an >> avro-collector.**** >> >> **** >> >> If you run more than one collector you can setup sink groups and define >> that it does failover or load balancing.**** >> >> **** >> >> The concept of a flume master from flume 0.9.x does not exist on >> flume-ng. I personally use the node and collector configs in the same >> config file under a different agent name, and then keep them synced on all >> machines. **** >> >> **** >> >> These two docs are pretty helpful:**** >> >> >> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst >> >> https://github.com/apache/flume/blob/trunk/flume-ng-doc/sphinx/FlumeDeveloperGuide.rst >> **** >> >> **** >> >> Thanks,**** >> >> **** >> >> Roy**** >> >> **** >> >> **** >> >> **** >> >> **** >> >> *From:* Juan Gentile [mailto:[email protected]] >> *Sent:* Tuesday, October 09, 2012 11:04 AM >> *To:* [email protected] >> *Subject:* Flume-ng - Distributed**** >> >> **** >> >> Hi,**** >> >> **** >> >> I'm new to Flume-ng, I'd like to ask you if you can tell me how I can >> accomplish to have an agent distributed in a cluster. I've have developed >> my own source and sink version that reads from a queue and the sink stores >> the messages read to hdfs. If I want to have this running in multiple >> instances, do I have to submit it on each node?**** >> >> **** >> >> This is my conf file:**** >> >> agent1.channels.channel1.type = memory**** >> >> agent1.channels.channel1.capacity = 1000**** >> >> agent1.channels.channel1.transactionCapacity = 1000**** >> >> **** >> >> agent1.sources.source1.channels = channel1**** >> >> agent1.sources.source1.type = MySource**** >> >> **** >> >> agent1.sinks.sink1.channel = channel1**** >> >> agent1.sinks.sink1.type = MySink**** >> >> **** >> >> agent1.channels = channel1**** >> >> agent1.sources = source1**** >> >> agent1.sinks = sink1**** >> >> **** >> >> **** >> >> I see that there is the concept of 'master' a 'node' in the previous >> version of flume, do I have something similar here?**** >> >> **** >> >> Thanks,**** >> >> Juan**** >> >> ** ** >> >> ** ** >> > >
