Thanks James, I took a shot at it today. It looks like everything I did worked, but I won’t be sure until I fix the storm slots issue. It would be good to know why things had to be configured as opposed to just winging it.
-- On September 28, 2016 at 17:40:15, James Sirota ([email protected]) wrote: We'll try to throw some docs together tomorrow to help you out. We need to make sure we are clear enough in our docs so everyone knows how to do what you are trying to do. We'll put more clarity around creating monit scripts and doing HDFS and ES pruning. 28.09.2016, 06:54, "Otto Fowler" <[email protected]>: > I’ve written the rules in the configuration, I’m going through the parser > deployment integration that I know about ( configuration for parser, > enrichment, es-templates ), but I still have to do the monit stuff, hdfs > pruning etc that I’ve been asking about > > -- > > Sent with Airmail > > On September 28, 2016 at 01:02:56, James Sirota ([email protected]) wrote: > > Do you still need help? Or were you able to do what you wanted? > > 27.09.2016, 12:34, "Otto Fowler" <[email protected]>: >> I’ll get there >> >> -- >> >> On September 27, 2016 at 15:23:10, Casey Stella ([email protected]) > > wrote: >> Yeah, it's confusing, so I feel your pain. I made a "Navigating the >> Architecture" (which should probably be named "Navigating the Developer >> Docs" ;) section in the top-level doc here >> <https://github.com/apache/incubator-metron#navigating-the-architecture> >> that >> links to the docs for various parts of the architecture. >> >> Hope that's helpful in the future! :) >> >> On Tue, Sep 27, 2016 at 3:05 PM, Otto Fowler <[email protected]> >> wrote: >> >>> Ok, I was going by the readme in metron-common and the wiki. >>> I’ll look in both readme files next time >>> >>> -- >>> >>> Sent with Airmail >>> >>> On September 27, 2016 at 14:49:29, Casey Stella ([email protected]) >>> wrote: >>> >>> It's described in > > https://github.com/apache/incubator-metron/tree/master/ >>> metron-platform/metron-parsers#fieldtransformation-configuration under >>> "REMOVE". The wiki tends to lag; the README.md docs are mostly >> >> up-to-date. >>> Casey >>> >>> On Tue, Sep 27, 2016 at 2:47 PM, Otto Fowler <[email protected]> >>> wrote: >>> >>> > I’m going by the wiki and REMOVE is not in there. Where should I look >> >> for >>> > the most up to date stuff? The unit tests? >>> > >>> > >>> > >>> > On September 27, 2016 at 14:33:08, Casey Stella ([email protected]) >>> > wrote: >>> > >>> > Yes, it is in the parser. You can assign a field to NULL and it will >>> > remove it. >>> > >>> > Consider the following example parser config which takes the foo > > field >>> and >>> > renames it to bar: >>> > >>> > { >>> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser", >>> > "sensorTopic":"custom", >>> > "parserConfig" : { >>> > "mapStrategy" : "UNFOLD" >>> > },"fieldTransformations" : [ >>> > { >>> > "transformation" : "STELLAR" >>> > ,"output" : [ "bar", "foo" ] >>> > ,"config" : { >>> > "bar" : "foo" >>> > ,"foo" : "NULL" >>> > } >>> > } >>> > ] >>> > } >>> > >>> > Note the output section, that defines the order in which the stellar >>> > statements are executed. You first set bar to whatever is in foo. > > Then >>> > you set foo to NULL, which will delete it. You will be left with just >>> bar, >>> > which will contain what is in foo. >>> > >>> > Another way to do this is to use the REMOVE transformation >>> > < >>> > https://github.com/apache/incubator-metron/tree/master/ >>> > metron-platform/metron-parsers#fieldtransformation-configuration> >>> > >>> > in >>> > combination with the stellar transformation: >>> > { >>> > "parserClassName":"org.apache.metron.parsers.json.JSONMapParser", >>> > "sensorTopic":"custom", >>> > "parserConfig" : { >>> > "mapStrategy" : "UNFOLD" >>> > },"fieldTransformations" : [ >>> > { >>> > "transformation" : "STELLAR" >>> > ,"output" : [ "bar", "foo" ] >>> > ,"config" : { >>> > "bar" : "foo" >>> > } >>> > }, >>> > { >>> > "input" : "foo" >>> > , "transformation" : "REMOVE" >>> > } >>> > ] >>> > } >>> > >>> > Best, >>> > >>> > Casey >>> > >>> > On Tue, Sep 27, 2016 at 2:12 PM, Otto Fowler < [email protected]> > >>> > wrote: >>> > >>> > > Is it possible to remove and rename fields with stellar? >>> > > I have “foreign” json coming out of the JSONParser, and I need to >>> rename >>> > > some fields to the core metron json object names and possibly > > remove >>> some >>> > > other fields. >>> > > >>> > > > ------------------- > Thank you, > > James Sirota > PPMC- Apache Metron (Incubating) > jsirota AT apache DOT org ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
