It seems quite easier than dealing with commons configuration. I might take a shot at it during the week-end, if I find the time.
About the customisation of the Application class, that's something which is already mostly done by https://issues.apache.org/jira/browse/FLUME-1491 : Dynamic Configuration from Zookeeper watcher It might be a good idea to split this improvement into : 1) Selection of an alternative configuration provider 2) A zookeeper based configuration provider If point 1 is done, anyone could use her/his custom configuration provider by dropping the implementation as a third party jar and then selecting it with the help of the command line. Bertrand On Fri, Mar 29, 2013 at 4:46 PM, Bertrand Dechoux <[email protected]>wrote: > Thanks for the hints! > > The part that you quoted is in fact from the flume wiki and I forgot to > quote it myself. But the page is quite old so I wasn't sure about the state > of the feature. I thought maybe commons-configuration was used but wasn't > be able to confirm it clearly (or more exactly I lost myself in this > dependency, which might be used by one of my current projects somehow). > > https://cwiki.apache.org/confluence/display/FLUME/Flume+NG > > Configuration Provider >> >> Flume NG has a pluggable configuration system called the configuration >> provider. By default, Flume NG ships with a Java property file based >> configuration system that is both simple and easy to generate >> programmatically. Flume OG has a centralized configuration system with >> a master and ZooKeeper for coordination and we recognize this is very >> appealing to some users where as others see it as overhead they simply >> don't want. We opted to make this a pluggable extension point and ship >> a basic implementation that would let many users get started quickly >> and easily. There's almost certainly enough desire for a similar >> implementation to that of Flume OG, but it isn't yet implemented. >> Users may also implement arbitrary plugins to integrate with any type >> of configuration system (JSON files, a shared RDBMS, a central system >> with ZooKeeper, and so forth). We see this as something more >> interesting to system integrators. >> > > I will look at the classes you pointed for my own education and I will see > what I can do. > > Thanks a lot. > > Bertrand > > > On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[email protected]> wrote: > >> Hi, >> >> "Flume NG has a pluggable configuration system called the configuration >> provider." >> >> There is an object hierarchy, but I don't consider it pluggable as there >> is no place to "plugin" your own implementation. I consider these things >> "private" to Flume. Meaning that there is no guaranteed API stability >> between releases. With that caveat, you'd want to implement the following >> abstract class: >> >> >> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java >> >> e.g >> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java >> >> and then'd you have to customize the Application class to "plugin" your >> own implementation: >> >> >> https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java >> >> Brock >> >> >> On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[email protected]>wrote: >> >>> Hi, >>> >>> Flume NG has a pluggable configuration system called the configuration >>> provider. I cloned the source repository and I indeed found it but I am a >>> bit lost. Is there some kind of short guide explaining how to write a >>> custom configuration provider? >>> >>> I would like to start with something simple : using a YAML configuration >>> file instead of a properties file. >>> It should be almost trivial to change the following configuration >>> >>> *agent1.channels.ch1.type = memory >>> >>> agent1.sources.avro-source1.channels = ch1 >>> agent1.sources.avro-source1.type = avro >>> agent1.sources.avro-source1.bind = 0.0.0.0 >>> agent1.sources.avro-source1.port = 41414 >>> >>> agent1.sinks.log-sink1.channel = ch1 >>> agent1.sinks.log-sink1.type = logger >>> >>> agent1.channels = ch1 >>> agent1.sources = avro-source1 >>> agent1.sinks = log-sink1* >>> >>> into >>> >>> agent1: >>> >>> channels: >>> ch1: >>> type: memory >>> >>> sources: >>> avroSource1: >>> channels: ch1 >>> type: avro >>> bind: 0.0.0.0 >>> port: 41414 >>> >>> sinks: >>> logSink1: >>> channel: ch1 >>> type: logger >>> >>> >>> At least, I know how to convert the latter into the former but I am not >>> sure how this transformation can be included into flume-ng using a >>> configuration provider. >>> I have not been able to find much information about the customisation of >>> the configuration provider. Did I miss any documentation? >>> >>> >>> Bertrand >>> >> >> >> >> -- >> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org >> > > -- Bertrand Dechoux
