Re: Parser Docs
Well, you do at the moment… I’m still keen on the idea of parsers emitting a schema of sorts, and then the framework being expanded to use that schema, along with enrichment schema and stellar output type inference to generate ES templates, but that’s another story. ES Template specs certainly do not belong in parsers, because they have to account for Enrichment, Intel and Triage data as well, which are not known to the parser stage. Agreed though, right now a working knowledge of ES templates is useful for parser dev. Simon > On 11 May 2017, at 14:51, Otto Fowler <ottobackwa...@gmail.com> wrote: > > I missed elasticsearch, you do need to understand ES indexing to setup the > correct storage etc for the fields you produce. > > > > On May 11, 2017 at 09:48:10, Otto Fowler (ottobackwa...@gmail.com > <mailto:ottobackwa...@gmail.com>) wrote: > >> Part of the point of having a framework like metron is that you don’t *need* >> to know those things to contribute parsers. >> What you would want to understand are the things at parser scope: >> >> STELLAR, >> the MessageParser interface >> The base ‘typed’ parsers ( JSONMap, CSVParser, BasicGrokParser ) >> >> and your data. >> >> Even the integration testing framework has been done in such a way that you >> can test with indexing, enrichment etc without having to worry >> about setting up zookeeper, kafka, storm etc. >> >> >> On May 11, 2017 at 09:38:46, Mark de Rijk >> (mark.der...@samarkconsulting.co.uk >> <mailto:mark.der...@samarkconsulting.co.uk>) wrote: >> >>> Hi, >>> >>> I really would like to help with parser development. I am reading up on >>> Nifi, Kafka and there are probably other topics I am missing now. >>> As when it comes to Hadoop ecosystem I am pretty much a beginner I will >>> unfortunately need a month to learn the different components. >>> >>> Regards, >>> Mark de Rijk >>> >>> >>> >>> Sent from my iPhone >>> >>> > On 9 May 2017, at 02:23, Otto Fowler <ottobackwa...@gmail.com >>> > <mailto:ottobackwa...@gmail.com>> wrote: >>> > >>> > There is a readme.md <http://readme.md/> PER parser in 777. >>> > I only stubbed them out however. >>> > >>> > Each parser created by the archetype has one as well. >>> > >>> > What I was hoping to do was to include the parser docs in the package >>> > assembly so the UI could load it. >>> > >>> > >>> > >>> > On May 8, 2017 at 19:35:41, Simon Elliston Ball >>> > (si...@simonellistonball.com <mailto:si...@simonellistonball.com>) wrote: >>> > >>> > Quick thought, and please shoot me down if this has already been thought >>> > of, but…. >>> > >>> > Would it be worthwhile if we put some per parser docs into the repo, >>> > essentially a README.md per major parser class, which would in theory be >>> > picked up by the docbook? We don’t really have much documentation right >>> > now on the effect of parserConfig settings for example. >>> > >>> > Similarly, is this something worth considering in the package format, or >>> > at least in the archetype proposed by METRON-777 (paging Ottto!) >>> > >>> > Worth the effort? I’m happy to do at least a few of the more recent ones >>> > I’ve spent meaningful time with. >>> > >>> > Simon
Re: Parser Docs
I missed elasticsearch, you do need to understand ES indexing to setup the correct storage etc for the fields you produce. On May 11, 2017 at 09:48:10, Otto Fowler (ottobackwa...@gmail.com) wrote: Part of the point of having a framework like metron is that you don’t *need* to know those things to contribute parsers. What you would want to understand are the things at parser scope: STELLAR, the MessageParser interface The base ‘typed’ parsers ( JSONMap, CSVParser, BasicGrokParser ) and your data. Even the integration testing framework has been done in such a way that you can test with indexing, enrichment etc without having to worry about setting up zookeeper, kafka, storm etc. On May 11, 2017 at 09:38:46, Mark de Rijk ( mark.der...@samarkconsulting.co.uk) wrote: Hi, I really would like to help with parser development. I am reading up on Nifi, Kafka and there are probably other topics I am missing now. As when it comes to Hadoop ecosystem I am pretty much a beginner I will unfortunately need a month to learn the different components. Regards, Mark de Rijk Sent from my iPhone > On 9 May 2017, at 02:23, Otto Fowler <ottobackwa...@gmail.com> wrote: > > There is a readme.md PER parser in 777. > I only stubbed them out however. > > Each parser created by the archetype has one as well. > > What I was hoping to do was to include the parser docs in the package assembly so the UI could load it. > > > > On May 8, 2017 at 19:35:41, Simon Elliston Ball ( si...@simonellistonball.com) wrote: > > Quick thought, and please shoot me down if this has already been thought of, but…. > > Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example. > > Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!) > > Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with. > > Simon
Re: Parser Docs
Part of the point of having a framework like metron is that you don’t *need* to know those things to contribute parsers. What you would want to understand are the things at parser scope: STELLAR, the MessageParser interface The base ‘typed’ parsers ( JSONMap, CSVParser, BasicGrokParser ) and your data. Even the integration testing framework has been done in such a way that you can test with indexing, enrichment etc without having to worry about setting up zookeeper, kafka, storm etc. On May 11, 2017 at 09:38:46, Mark de Rijk ( mark.der...@samarkconsulting.co.uk) wrote: Hi, I really would like to help with parser development. I am reading up on Nifi, Kafka and there are probably other topics I am missing now. As when it comes to Hadoop ecosystem I am pretty much a beginner I will unfortunately need a month to learn the different components. Regards, Mark de Rijk Sent from my iPhone > On 9 May 2017, at 02:23, Otto Fowler <ottobackwa...@gmail.com> wrote: > > There is a readme.md PER parser in 777. > I only stubbed them out however. > > Each parser created by the archetype has one as well. > > What I was hoping to do was to include the parser docs in the package assembly so the UI could load it. > > > > On May 8, 2017 at 19:35:41, Simon Elliston Ball ( si...@simonellistonball.com) wrote: > > Quick thought, and please shoot me down if this has already been thought of, but…. > > Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example. > > Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!) > > Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with. > > Simon
Re: Parser Docs
Hi Mark, If you’re looking to write Metron parsers you aren’t going to have to worry about Nifi or Kafka in any level of detail. The parser interface just gets byte[] and outputs JSON. Of course I would never recommend avoiding the reading around all the other exciting bits and components around the eco-system, but if you want to dive in sooner, Metron parsers are a nice easy and self-contained place to start. Simon > On 11 May 2017, at 14:38, Mark de Rijk <mark.der...@samarkconsulting.co.uk> > wrote: > > Hi, > > I really would like to help with parser development. I am reading up on Nifi, > Kafka and there are probably other topics I am missing now. > As when it comes to Hadoop ecosystem I am pretty much a beginner I will > unfortunately need a month to learn the different components. > > Regards, > Mark de Rijk > > > > Sent from my iPhone > >> On 9 May 2017, at 02:23, Otto Fowler <ottobackwa...@gmail.com> wrote: >> >> There is a readme.md PER parser in 777. >> I only stubbed them out however. >> >> Each parser created by the archetype has one as well. >> >> What I was hoping to do was to include the parser docs in the package >> assembly so the UI could load it. >> >> >> >> On May 8, 2017 at 19:35:41, Simon Elliston Ball >> (si...@simonellistonball.com) wrote: >> >> Quick thought, and please shoot me down if this has already been thought of, >> but…. >> >> Would it be worthwhile if we put some per parser docs into the repo, >> essentially a README.md per major parser class, which would in theory be >> picked up by the docbook? We don’t really have much documentation right now >> on the effect of parserConfig settings for example. >> >> Similarly, is this something worth considering in the package format, or at >> least in the archetype proposed by METRON-777 (paging Ottto!) >> >> Worth the effort? I’m happy to do at least a few of the more recent ones >> I’ve spent meaningful time with. >> >> Simon
Re: Parser Docs
Hi, I really would like to help with parser development. I am reading up on Nifi, Kafka and there are probably other topics I am missing now. As when it comes to Hadoop ecosystem I am pretty much a beginner I will unfortunately need a month to learn the different components. Regards, Mark de Rijk Sent from my iPhone > On 9 May 2017, at 02:23, Otto Fowler <ottobackwa...@gmail.com> wrote: > > There is a readme.md PER parser in 777. > I only stubbed them out however. > > Each parser created by the archetype has one as well. > > What I was hoping to do was to include the parser docs in the package > assembly so the UI could load it. > > > > On May 8, 2017 at 19:35:41, Simon Elliston Ball (si...@simonellistonball.com) > wrote: > > Quick thought, and please shoot me down if this has already been thought of, > but…. > > Would it be worthwhile if we put some per parser docs into the repo, > essentially a README.md per major parser class, which would in theory be > picked up by the docbook? We don’t really have much documentation right now > on the effect of parserConfig settings for example. > > Similarly, is this something worth considering in the package format, or at > least in the archetype proposed by METRON-777 (paging Ottto!) > > Worth the effort? I’m happy to do at least a few of the more recent ones I’ve > spent meaningful time with. > > Simon
Re: Parser Docs
There is a readme.md PER parser in 777. I only stubbed them out however. Each parser created by the archetype has one as well. What I was hoping to do was to include the parser docs in the package assembly so the UI could load it. On May 8, 2017 at 19:35:41, Simon Elliston Ball (si...@simonellistonball.com) wrote: Quick thought, and please shoot me down if this has already been thought of, but…. Would it be worthwhile if we put some per parser docs into the repo, essentially a README.md per major parser class, which would in theory be picked up by the docbook? We don’t really have much documentation right now on the effect of parserConfig settings for example. Similarly, is this something worth considering in the package format, or at least in the archetype proposed by METRON-777 (paging Ottto!) Worth the effort? I’m happy to do at least a few of the more recent ones I’ve spent meaningful time with. Simon
Re: Parser Docs
Definitely worthwhile. I discussed something similar (but more general) a little while back here <https://lists.apache.org/thread.html/17904a10b8f41a97dfac84dcb3c22a1a0229fa9db170fa762741c2fe@%3Cdev.metron.apache.org%3E>. Totally worth the effort IMO. Jon On Mon, May 8, 2017 at 7:36 PM Casey Stella <ceste...@gmail.com> wrote: > +1 for parser docs > > On Mon, May 8, 2017 at 7:35 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > > > Quick thought, and please shoot me down if this has already been thought > > of, but…. > > > > Would it be worthwhile if we put some per parser docs into the repo, > > essentially a README.md per major parser class, which would in theory be > > picked up by the docbook? We don’t really have much documentation right > now > > on the effect of parserConfig settings for example. > > > > Similarly, is this something worth considering in the package format, or > > at least in the archetype proposed by METRON-777 (paging Ottto!) > > > > Worth the effort? I’m happy to do at least a few of the more recent ones > > I’ve spent meaningful time with. > > > > Simon > -- Jon
