Re: Define and leverage custom constants in agent configuration

Fri, 02 Dec 2016 00:34:06 -0800 

That would be a cool option, Donat!


On 01/12/2016 19:41, Bessenyei Balázs Donát wrote:


Hi Roberto,
Do you mean something like the proposed change at https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your requirements? 

Thank you,

Donat
On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <[email protected] <mailto:[email protected]>> wrote: 

    Hi Robert,

    Looks like what you want is more like a global configuration, and
    based on my limited knowledge in Apache Flume, it is missing now.

    BR,

    Shang

    *From: *Roberto Coluccio <[email protected]
    <mailto:[email protected]>>
    *Reply-To: *"[email protected] <mailto:[email protected]>"
    <[email protected] <mailto:[email protected]>>
    *Date: *Thursday, December 1, 2016 at 2:59 AM
    *To: *"[email protected] <mailto:[email protected]>"
    <[email protected] <mailto:[email protected]>>
    *Subject: *Re: Define and leverage custom constants in agent
    configuration

    Hi Silvio,

    thanks for your suggestion. It gives me a workaround, but it's not
    exactly what I was hoping/asking about.

    I'm going to assume what I'd want to do being just not doable.

    Thank you,

    Roberto

    On 29/11/2016 09:26, Silvio Di gregorio wrote:

        with the "static interceptor" you can add key/value pairs in
        your flume event header:

        agentOne.sources.s1.interceptors = interOne

        agentOne.sources.s1.interceptors.interOne.type = static

        agentOne.sources.s1.interceptors.interOne.key = costante

        agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi

        I hope it is useful

        have nice day

        Silvio

        2016-11-28 18:26 GMT+01:00 Roberto Coluccio
        <[email protected] <mailto:[email protected]>>:

            Hello folks,

            I was wondering if it's possible to define custom
            constants in a Flume agent configuration file, so to avoid
            errors and replicated stuff when not needed. An example to
            make my point clear:

            assume I want to launch an agent of type /myAgent/
            configured with file channel and hdfs sink.

            I want the file channel to persist data and checkpoint
            into "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"

            Also, I want the hdfs sink to write files with prefix
            "myAgent1"

            In this scenario, I should harcorde the "myAgent1" string
            3 times, one for each configuration parameter. This is
            error prone and kinda frustrating.

            Is there a way to define a constant like "myConstant" in
            the configuration file, and leverage it as parameter in
            the other configuration lines?

            e.g.

            myConstant = myAgent1
            myAgent.channels.myChannel.dataDirs =
            ~/flume/${myConstant}/data
            myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}

            It seems like the use of flume events header keys, but I
            don't want to add interceptors or other components just to
            have parameters in my config file.

            Thanks for your help.

            Best regards,

            Roberto
-- 
    **

    **

    **

    **

    *Roberto Coluccio*

    Solution Designer
    Direzione Ricerca e Innovazione
    [email protected] <mailto:[email protected]>

    Interno 604135 | Esterno +39 0510435135 <tel:+39%20051%20043%205135>



    *Engineering Ingegneria Informatica spa*
    Via Marconi, 10, 40122 Bologna

    www.eng.it <http://www.eng.it/>



--

**

**

**

**

**

*Roberto Coluccio*

Solution Designer
Direzione Ricerca e Innovazione
[email protected] <mailto:[email protected]>

Interno 604135 | Esterno +39 0510435135



*Engineering Ingegneria Informatica spa*
Via Marconi, 10, 40122 Bologna

www.eng.it <http://www.eng.it/>

Reply via email to