A couple things I would like to point out. You can test Stellar statements without having to send data through parser/enrichment topologies. There is a REST endpoint that allows you to pass in a sample message and parser config and returns a message with Stellar statements applied. This could easily be expanded to enrichment configs or testing generic stellar statements against test messages.
Moving statements to a separate file is going to require a lot of work and will make our mechanism for managing configuration in bolts more complex. We would have to also listen for changes in these files and reconcile which parser/enrichment configs are affected. On Fri, Jul 14, 2017 at 12:42 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > I think the ‘files’ should be stored in zk, and updated with the same > mechanism. > > On July 14, 2017 at 13:27:36, Matt Foley (mfo...@hortonworks.com) wrote: > > In the abstract, this is a good idea. I see it as related to METRON-987, > which was the first step in allowing sequences of Stellar statements (aka > "programs" :-) ) instead of just unrelated groups of single statements. > Your proposal lets us really work with programs as first-class entities. > > However, some concerns need to be resolved: > > 1. Syntax. > > Currently Stellar syntax and JSON fit neatly together. Where would be the > cut line for file substitutions? Referencing METRON-987, would you only > allow a file substitution where we currently allow square-bracketed Stellar > string sequences? What about Profile config syntax, where several chunks of > code are intimately related (hence want to be located in the same file), > but don't all get executed at the same time? (This is not a showstopper > question because Profile configs are usually simple and don't really need > file substitution. The need is much greater in Enrichment.) > > 2. Config Updates. > > Currently Metron configuration is stored in ZK, but managed through Curator > libraries. In return for considerable complexity, this gives instant update > whenever a config changes, without effort in the BI part of the > application. This differs sharply from file-based configuration, where > updates in response to config changes require either a restart, an explicit > reload command from the user, or frequent state-checking in the > application. > > So currently people trying to develop a new enrichment can update the > config, and immediately test the result, without restarting and without any > explicit reload command. We probably want to not lose this. > > Rather than roll our own file pointer model, can we use JSON Pointers? Will > they work with Curator? Both of those get into some fairly obscure > features, that would need to be studied. It also actually relates to the > syntax question presented above. > > > On 7/14/17, 6:17 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: > > https://issues.apache.org/jira/browse/METRON-1046 > > I was thinking this morning that managing stellar statements in the config > json could become, and maybe is kind of unwieldy. > To that end, if in say a parser configuration I can refer to a ‘file’ in > zookeeper as an alternative, we would add the capability to execute and > manage more complex statements, and even chain multiple statements > together. > > These files could be shared as well. > > This could be a Bad Idea™, so I thought I’d throw it out to the list. > > Please check out the jira, give some thought, and comment there or on the > list or both. > > O >
