Hello folks,

I was wondering if it's possible to define custom constants in a Flume agent configuration file, so to avoid errors and replicated stuff when not needed. An example to make my point clear:

assume I want to launch an agent of type /myAgent/ configured with file channel and hdfs sink.

I want the file channel to persist data and checkpoint into "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"

Also, I want the hdfs sink to write files with prefix "myAgent1"

In this scenario, I should harcorde the "myAgent1" string 3 times, one for each configuration parameter. This is error prone and kinda frustrating.

Is there a way to define a constant like "myConstant" in the configuration file, and leverage it as parameter in the other configuration lines?

e.g.

myConstant = myAgent1
myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}

It seems like the use of flume events header keys, but I don't want to add interceptors or other components just to have parameters in my config file.

Thanks for your help.

Best regards,

Roberto


Reply via email to