GitHub user merrimanr opened a pull request:

    Grok patterns are now read from zookeeper parser config property 

    This PR moves Grok patterns from HDFS to a property in Zookeeper 
("SensorParserConfig.parserConfig.grokPattern").  Most of the important changes 
were made to the GrokParser class and related tests.  There were 2 primary 
challenges with implementing this:
    - A change to a Grok pattern in Zookeeper should take effect immediately 
without a topology restart.  Currently the MessageParser does not have access 
to the Zookeeper config.
    - Grok patterns must now comply with JSON formatting standards (files had 
less restrictions)
    The first challenge was solved by changing the MessageParser.parse method 
to include an additional "SensorParserConfig" parameter that the GrokParser can 
use to determine if the grok pattern has changed and reinit if necessary.  The 
ParserBolt now passes in the most recent SensorParserConfig to the 
MessageParser for every message processed.  This is similar to how the writer 
interfaces work.  I believe this approach is simpler than having the ParserBolt 
listen for changes and reconfigure/reinit the GrokParser.  Changing this 
interface makes this PR look much bigger than it actually is because all the 
parsers and parser tests had to be adjusted.  Most of files included are 1 line 
    The second challenge was overcome by using '\n' to separate lines inside 
the grok pattern JSON property.  An extra '\' also had to be added to '\' 
escape characters in the grok pattern files.  The advantage of this is that no 
special serialization/deserialization is needed inside of the GrokParser code.  
The disadvantage is that the patterns are not as human-readable.
    All unit tests and integration tests are passing.  I added an extra unit 
test in SampleGrokParserTest to simulate and test a config being updated.  I 
also tested yaf and squid sensors in quick-dev by updating the grokPattern 
properties and verifying the new patterns were applied. 

You can merge this pull request into a Git repository by running:

    $ git pull METRON-498

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #308
commit 56be7560dd42fefadfb0d88691b76993943da449
Author: rmerriman <>
Date:   2016-10-12T18:46:45Z

    Grok patterns are now read from zookeeper parser config property 


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

Reply via email to