Hi Bertrand, https://issues.apache.org/jira/browse/FLUME-1630 changed a lot of the config system, and made it much easier to write a config provider. Once you write a provider you can update the application class to use reflection to instantiate the provider. I dont remember how the patch on FLUME-1491 looks like, but if it is possible to integrate it into this, it might make sense.
Thanks Hari -- Hari Shreedharan On Tuesday, April 2, 2013 at 12:23 PM, Bertrand Dechoux wrote: > I did a short proof of concept for the YAML part here > https://github.com/BertrandDechoux/flume-ng-yaml > > Does your reaction mean that FLUME-1491 is a 'zombie'? Anyway, I will look at > what was done in that issue and see what can be done for a clean selection of > a custom configuration provider. > > Bertrand > > > On Fri, Mar 29, 2013 at 8:06 PM, Hari Shreedharan <[email protected] > (mailto:[email protected])> wrote: > > I actually do like the idea of pluggable config provider. Like you > > mentioned, it should be possible to hack the Application class, and use a > > CLI arg to select the config provider. If no CLI arg is provided, just fall > > back to the current mechanism. If you submit a patch, I'd be glad to review > > it :-) > > > > > > Thanks > > Hari > > > > -- > > Hari Shreedharan > > > > > > On Friday, March 29, 2013 at 9:43 AM, Bertrand Dechoux wrote: > > > > > It seems quite easier than dealing with commons configuration. > > > > > > I might take a shot at it during the week-end, if I find the time. > > > > > > About the customisation of the Application class, that's something which > > > is already mostly done by > > > https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration > > > from Zookeeper watcher > > > > > > It might be a good idea to split this improvement into : > > > 1) Selection of an alternative configuration provider > > > 2) A zookeeper based configuration provider > > > > > > If point 1 is done, anyone could use her/his custom configuration > > > provider by dropping the implementation as a third party jar and then > > > selecting it with the help of the command line. > > > > > > Bertrand > > > > > > > > > On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[email protected] > > > (mailto:[email protected])> wrote: > > > > Thanks for the hints! > > > > > > > > The part that you quoted is in fact from the flume wiki and I forgot to > > > > quote it myself. But the page is quite old so I wasn't sure about the > > > > state of the feature. I thought maybe commons-configuration was used > > > > but wasn't be able to confirm it clearly (or more exactly I lost myself > > > > in this dependency, which might be used by one of my current projects > > > > somehow). > > > > > > > > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG > > > > > > > > > Configuration Provider > > > > > > > > > > Flume NG has a pluggable configuration system called the configuration > > > > > provider. By default, Flume NG ships with a Java property file based > > > > > configuration system that is both simple and easy to generate > > > > > programmatically. Flume OG has a centralized configuration system with > > > > > a master and ZooKeeper for coordination and we recognize this is very > > > > > appealing to some users where as others see it as overhead they simply > > > > > don't want. We opted to make this a pluggable extension point and ship > > > > > a basic implementation that would let many users get started quickly > > > > > and easily. There's almost certainly enough desire for a similar > > > > > implementation to that of Flume OG, but it isn't yet implemented. > > > > > Users may also implement arbitrary plugins to integrate with any type > > > > > of configuration system (JSON files, a shared RDBMS, a central system > > > > > with ZooKeeper, and so forth). We see this as something more > > > > > interesting to system integrators. > > > > > > > > I will look at the classes you pointed for my own education and I will > > > > see what I can do. > > > > > > > > Thanks a lot. > > > > > > > > Bertrand > > > > > > > > > > > > On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[email protected] > > > > (mailto:[email protected])> wrote: > > > > > Hi, > > > > > > > > > > "Flume NG has a pluggable configuration system called the > > > > > configuration provider." > > > > > > > > > > There is an object hierarchy, but I don't consider it pluggable as > > > > > there is no place to "plugin" your own implementation. I consider > > > > > these things "private" to Flume. Meaning that there is no guaranteed > > > > > API stability between releases. With that caveat, you'd want to > > > > > implement the following abstract class: > > > > > > > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java > > > > > > > > > > e.g > > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java > > > > > > > > > > > > > > > and then'd you have to customize the Application class to "plugin" > > > > > your own implementation: > > > > > > > > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java > > > > > > > > > > > > > > > Brock > > > > > > > > > > > > > > > On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[email protected] > > > > > (mailto:[email protected])> wrote: > > > > > > Hi, > > > > > > > > > > > > Flume NG has a pluggable configuration system called the > > > > > > configuration provider. I cloned the source repository and I indeed > > > > > > found it but I am a bit lost. Is there some kind of short guide > > > > > > explaining how to write a custom configuration provider? > > > > > > > > > > > > I would like to start with something simple : using a YAML > > > > > > configuration file instead of a properties file. > > > > > > It should be almost trivial to change the following configuration > > > > > > > > > > > > agent1.channels.ch1.type = memory > > > > > > > > > > > > agent1.sources.avro-source1.channels = ch1 > > > > > > agent1.sources.avro-source1.type = avro > > > > > > agent1.sources.avro-source1.bind = 0.0.0.0 > > > > > > agent1.sources.avro-source1.port = 41414 > > > > > > > > > > > > agent1.sinks.log-sink1.channel = ch1 > > > > > > agent1.sinks.log-sink1.type = logger > > > > > > > > > > > > agent1.channels = ch1 > > > > > > agent1.sources = avro-source1 > > > > > > agent1.sinks = log-sink1 > > > > > > > > > > > > into > > > > > > > > > > > > agent1: > > > > > > > > > > > > channels: > > > > > > ch1: > > > > > > type: memory > > > > > > > > > > > > sources: > > > > > > avroSource1: > > > > > > channels: ch1 > > > > > > type: avro > > > > > > bind: 0.0.0.0 > > > > > > port: 41414 > > > > > > > > > > > > sinks: > > > > > > logSink1: > > > > > > channel: ch1 > > > > > > type: logger > > > > > > > > > > > > > > > > > > At least, I know how to convert the latter into the former but I am > > > > > > not sure how this transformation can be included into flume-ng > > > > > > using a configuration provider. > > > > > > I have not been able to find much information about the > > > > > > customisation of the configuration provider. Did I miss any > > > > > > documentation? > > > > > > > > > > > > > > > > > > Bertrand > > > > > > > > > > > > > > > > > > > > -- > > > > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org > > > > > > > > > > > > -- > > > Bertrand Dechoux > > > > > > -- > Bertrand Dechoux
