[
https://issues.apache.org/jira/browse/FLUME-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601726#comment-13601726
]
Mike Percy commented on FLUME-1941:
-----------------------------------
Gabriel, thanks a lot for the feedback!
I like the recursive concept. Of course, the impl will have to watch out for
self-loops.
I also think defining variables is a great idea. Due to recent changes that
went into the exec source to support arbitrary shell syntax, I'm concerned
about conflicting with shell variables however. So I think we would need to use
non-bash syntax for the substitution. Maybe something like JIRA
double-brackets? (this is basically JIRA syntax, so watch out when posting it
in here. I'm putting all the syntax into JIRA "noformat" sections from here on
out):
{noformat}
agent.sink.sink-1.foo = /bar/{{variable-name}}/baz
{noformat}
Maybe someone who is more familiar with Bash than I am would like to provide
feedback on this issue? Note also that the BucketPath stuff in HDFS sink has
some similar-looking and magical date- and variable-substitution syntax which
looks like:
{noformat}
agent.sink.sink-1.path = /path/to/hdfs/%{header-name}/%YY-%mm-%dd/%HH:%MM/blah
{noformat}
It's best to restrict the variable definitions to be constrained to some type
of hierarchy as well, so we maintain the flexibility of extending the syntax as
needed in the future. Maybe something like:
{noformat}
agent1.__vars__.my-variable-name = foo
{noformat}
as well as
{noformat}
__global__.__vars__.my-variable-name = bar
{noformat}
We could easily get into ambiguities during parsing by allowing dots in
variable names, so I'd advocate against that.
Thoughts? Alternative suggestions most welcome.
> Support defaults or inheritance in configs
> ------------------------------------------
>
> Key: FLUME-1941
> URL: https://issues.apache.org/jira/browse/FLUME-1941
> Project: Flume
> Issue Type: New Feature
> Components: Configuration
> Reporter: Mike Percy
> Fix For: v1.4.0
>
>
> Proposal to support defaults or inheritance in configs.
> The idea is to create a "prototypal" component config, such as a source or
> sink, which is not necessarily instantiated but is used to avoid repetitive
> configurations when creating multiple components of the same type. A great
> example of this is users who define 5 HDFS sinks to increase write
> parallelism, but they each contain many of the same configuration parameters
> and differ only in their name and path.
> Basic idea:
> {noformat}
> agent.sinks.sink-proto-1.type = my-sink
> agent.sinks.sink-proto-1.path = /var/log/foo/bar
> agent.sinks.sink-proto-1.serializer = MySerializer$Builder
> agent.sinks.sink-proto-1.credentials = mpercy
> agent.sinks.sink-1.__prototype__ = sink-proto-1
> agent.sinks.sink-1.path = /var/log/baz/blam
> agent.sinks.sink-2.__prototype__ = sink-proto-1
> agent.sinks.sink-2.path = /var/log/glerp/bazinga
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
