Having a utility to test ingest of a log format is very helpful. For example having a UI where you can start with a log and then work through the resulting events. A unit testing framework is helpful too for supporting upgrades and validating new versions of the logs. For example a repository of log samples plus their known good results. Run the logs through grok and verify that the results are correct.
Thanks Carolyn On 9/25/16, 10:30 PM, "James Sirota" <[email protected]> wrote: >I think having the ability to validate Grok statements from within the REPL is >useful. It would save the user the manual step of pasting Grok expressions >and sample logs into Grok validators. Also, if we can potentially embed grok >validators into a map-only job we can instantly see what % of our corpus the >expression can parse and can get a dump of every log line it cannot parse. So >I think you are on to something here. > >James > >25.09.2016, 07:40, "Otto Fowler" <[email protected]>: >> Casey, >> >> Would things implemented this way be a better way to automate these flows ( >> built in error handling and checking as standards ) and not just for a user >> at the terminal? >> >> On September 24, 2016 at 19:02:59, Casey Stella ([email protected]) wrote: >> >> Sorry, I meant to ask if anyone had feedback or thoughts. >> On Sat, Sep 24, 2016 at 18:35 Casey Stella <[email protected]> wrote: >> >>> Grok-based parsers are an important part of Metron and Grok is an >>> extremely adaptable, if complex, language for specifying structure in >> >> data. >>> Right now, the flow is generally: >>> >>> 1. Get some sample data >>> 2. Use an online utility to test grok statements >>> 3. upload grok statement to HDFS >>> 4. restart parser topology >>> >>> Given the demo around managing stellar transformations inside of the >> >> REPL, >>> I began to think that it might be a more coherent and integrated >> >> experience >>> to manage grok inside of the REPL. >>> >>> This would involve stellar functions to >>> >>> - test grok statements >>> - read grok statements from HDFS >>> - push grok statements to HDFS >>> >>> I made a few JIRAs to outline it here >>> <https://issues.apache.org/jira/browse/METRON-454>. > >------------------- >Thank you, > >James Sirota >PPMC- Apache Metron (Incubating) >jsirota AT apache DOT org >
