In the abstract, this is a good idea. I see it as related to METRON-987, which
was the first step in allowing sequences of Stellar statements (aka "programs"
:-) ) instead of just unrelated groups of single statements. Your proposal
lets us really work with programs as first-class entities.
However, some concerns need to be resolved:
1. Syntax.
Currently Stellar syntax and JSON fit neatly together. Where would be the cut
line for file substitutions? Referencing METRON-987, would you only allow a
file substitution where we currently allow square-bracketed Stellar string
sequences? What about Profile config syntax, where several chunks of code are
intimately related (hence want to be located in the same file), but don't all
get executed at the same time? (This is not a showstopper question because
Profile configs are usually simple and don't really need file substitution.
The need is much greater in Enrichment.)
2. Config Updates.
Currently Metron configuration is stored in ZK, but managed through Curator
libraries. In return for considerable complexity, this gives instant update
whenever a config changes, without effort in the BI part of the application.
This differs sharply from file-based configuration, where updates in response
to config changes require either a restart, an explicit reload command from the
user, or frequent state-checking in the application.
So currently people trying to develop a new enrichment can update the config,
and immediately test the result, without restarting and without any explicit
reload command. We probably want to not lose this.
Rather than roll our own file pointer model, can we use JSON Pointers? Will
they work with Curator? Both of those get into some fairly obscure features,
that would need to be studied. It also actually relates to the syntax question
presented above.
On 7/14/17, 6:17 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:
https://issues.apache.org/jira/browse/METRON-1046
I was thinking this morning that managing stellar statements in the config
json could become, and maybe is kind of unwieldy.
To that end, if in say a parser configuration I can refer to a ‘file’ in
zookeeper as an alternative, we would add the capability to execute and
manage more complex statements, and even chain multiple statements together.
These files could be shared as well.
This could be a Bad Idea™, so I thought I’d throw it out to the list.
Please check out the jira, give some thought, and comment there or on the
list or both.
O
