Thanks for the hints! The part that you quoted is in fact from the flume wiki and I forgot to quote it myself. But the page is quite old so I wasn't sure about the state of the feature. I thought maybe commons-configuration was used but wasn't be able to confirm it clearly (or more exactly I lost myself in this dependency, which might be used by one of my current projects somehow).
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG Configuration Provider > Flume NG has a pluggable configuration system called the configuration > provider. By default, Flume NG ships with a Java property file based > configuration system that is both simple and easy to generate > programmatically. Flume OG has a centralized configuration system with > a master and ZooKeeper for coordination and we recognize this is very > appealing to some users where as others see it as overhead they simply > don't want. We opted to make this a pluggable extension point and ship > a basic implementation that would let many users get started quickly > and easily. There's almost certainly enough desire for a similar > implementation to that of Flume OG, but it isn't yet implemented. > Users may also implement arbitrary plugins to integrate with any type > of configuration system (JSON files, a shared RDBMS, a central system > with ZooKeeper, and so forth). We see this as something more > interesting to system integrators. > I will look at the classes you pointed for my own education and I will see what I can do. Thanks a lot. Bertrand On Fri, Mar 29, 2013 at 4:19 PM, Brock Noland <[email protected]> wrote: > Hi, > > "Flume NG has a pluggable configuration system called the configuration > provider." > > There is an object hierarchy, but I don't consider it pluggable as there > is no place to "plugin" your own implementation. I consider these things > "private" to Flume. Meaning that there is no guaranteed API stability > between releases. With that caveat, you'd want to implement the following > abstract class: > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java > > e.g > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/PropertiesFileConfigurationProvider.java > > and then'd you have to customize the Application class to "plugin" your > own implementation: > > > https://github.com/apache/flume/blob/trunk/flume-ng-node/src/main/java/org/apache/flume/node/Application.java > > Brock > > > On Fri, Mar 29, 2013 at 8:53 AM, Bertrand Dechoux <[email protected]>wrote: > >> Hi, >> >> Flume NG has a pluggable configuration system called the configuration >> provider. I cloned the source repository and I indeed found it but I am a >> bit lost. Is there some kind of short guide explaining how to write a >> custom configuration provider? >> >> I would like to start with something simple : using a YAML configuration >> file instead of a properties file. >> It should be almost trivial to change the following configuration >> >> *agent1.channels.ch1.type = memory >> >> agent1.sources.avro-source1.channels = ch1 >> agent1.sources.avro-source1.type = avro >> agent1.sources.avro-source1.bind = 0.0.0.0 >> agent1.sources.avro-source1.port = 41414 >> >> agent1.sinks.log-sink1.channel = ch1 >> agent1.sinks.log-sink1.type = logger >> >> agent1.channels = ch1 >> agent1.sources = avro-source1 >> agent1.sinks = log-sink1* >> >> into >> >> agent1: >> >> channels: >> ch1: >> type: memory >> >> sources: >> avroSource1: >> channels: ch1 >> type: avro >> bind: 0.0.0.0 >> port: 41414 >> >> sinks: >> logSink1: >> channel: ch1 >> type: logger >> >> >> At least, I know how to convert the latter into the former but I am not >> sure how this transformation can be included into flume-ng using a >> configuration provider. >> I have not been able to find much information about the customisation of >> the configuration provider. Did I miss any documentation? >> >> >> Bertrand >> > > > > -- > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org >
