Re: [DISCUSS] Contributing a General Purpose Regex Parser
+1
This looks like it would be a great contribution. It might be worth having
a look at this in the context of REGEX_ROUTING, which does a similar thing,
but requires a very large number of disparate sensor configs. In that
context, I would say this provides a good means of handling things like
server syslog sources in particular.
It would be great to see a JIRA and PR on this. Discussion around any
configuration specifics is probably easier around some code.
Also, it would be really interesting to hear about any performance thoughts
between something like this vs a complex pattern in Grok for instance, or
the approach taken in the default ASA parser, which is really quite similar
to this, but more 'coded in'.
Simon
On Mon, 27 Aug 2018 at 11:28, wrote:
> Hello,
>
>
>
> We have implemented a general purpose regex parser for Metron that we are
> interested in contributing back to the community.
>
>
>
> While the Metron Grok parser provides some regex based capability today,
> the intention of this general purpose regex parser is to:
>
>1. Allow for more advanced parsing scenarios (specifically, dealing with
>multiple regex lines for devices that contain several log formats within
>them)
>2. Give users and developers of Metron additional options for parsing
>3. With the new parser chaining and regex routing feature available in
>Metron, this gives some additional flexibility to logically separate a
> flow
>by:
> 1. Regex routing to segregate logs at a device level and handle
> envelope unwrapping
> 2. This general purpose regex parser to parse an entire device type
> that contains multiple log formats within the single device (for
> example,
> RHEL logs)
>
>
>
> At a high level control flow is like this:
>
> 1. Identify the record type if incoming raw message.
>
> 2. Find and apply the regular expression of corresponding record type to
> extract the fields (using named groups).
>
> 3. Apply the message header regex to extract the fields in the header part
> of the message (using named groups).
>
>
> The parser config uses the following structure:
>
>"recordTypeRegex": "(?(?<=\\s)\\b(kernel|syslog)\\b(?=\\[|:))"
>
>"messageHeaderRegex": "(?(?<=^<)
>
> \\d{1,4}(?=>)).*?(?(?<=>)[A-Za-z]{3}\\s{1,2}\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2}(?=\\s)).*?(?(?<=\\s).*?(?=\\s))
> ",
>
>"fields": [
>
> {
>
> "recordType": "kernel",
>
> "regex": ".*(?(?<=\\]|\\w\\:).*?(?=$))"
>
> },
>
> {
>
> "recordType": "syslog",
>
> "regex":
>
> ".*(?(?<=PID\\s=\\s).*?(?=\\sLine)).*(?(?<=64\\s)\/([A-Za-z0-9_-]+\/)+(?=\\w))(?.*?(?=\")).*(?(?<=\").*?(?=$))"
>
> }
>
> ]
>
>
>
> Where:
>
>- recordTypeRegex is used to distinctly identify a record type. It
>inputs a valid regular expression and may also have named groups, which
>would be extracted into fields.
>- messageHeaderRegex is used to specify a regular expression to extract
>fields from a message part which is common across all the messages (i.e,
>syslog fields, standard headers)
>- fields: json list of objects containing recordType and regex. The
>expression that is evaluated is based on the output of the
> recordTypeRegex
>- Note: recordTypeRegex and messageHeaderRegex could be specified as
>lists also (as a JSON array), where the list will be evaluated in order
>until a matching regular expression is found.
>
>
>
>
>
> If there are no objections to having this type of Parser within Metron, we
> will open a JIRA/PR for code review.
>
> *Jagdeep Singh*
>
--
--
simon elliston ball
@sireb
Re: [DISCUSS] Getting to a 1.0 release
There is a jira to make indexing extensible, such that you could define the indexing topology much as you define the parsing, and can load different indexers. On August 27, 2018 at 02:44:24, Ali Nazemian (alinazem...@gmail.com) wrote: One thing that we could imagine for v1.0 might be an ability to extend Metron from adding more pipelines to it. For example, being able to extend Metron to be integrated with other endpoints more easily from Storm perspective. For example, what if we would like to create other topologies to write files in ORC directly rather than HDFS or index it to Druid. What if we would like to build an automatic security response and move to become SOAR. All these integrations can be done even now and there are other users that may have done it already. However, providing a clear extension point can make it easier to contribute other pipelines to the community. I think by adding this level of extendability, Metron community will grow way faster by adding more integration that can be available. Cheers, Ali On Mon, Aug 20, 2018 at 10:50 PM Casey Stella wrote: > I completely agree, Mike. Our docs are either very high level or very low > level (and possibly stale) and, worse, aren't aimed at the actors that > you've stated. > I think that the HBase project does a good job of providing coherent and > useable documentation in their "HBase Book" (see > https://hbase.apache.org/book.html). > It's not actor-specific, but it is coherent advice for the practical > practitioner of HBase (both admin and developer) and speaks with one > voice. I think Metron's need > is a bit different, but at the minimum some coherent docs that speaks with > one voice and has a coherent pitch about what Metron is used for and what > it isn't used for > is well needed. > > On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Apologies for any spelling mishaps as I'm writing from my phone. > > > > I'm for improving our docs. I'd like to see us guide our various profiles > > of user towards the specific documentation for the abstraction levels > > they'll be most interested in working from. I think we should have > platform > > docs about how we're a broadly useful, extensible streaming analytics > > platform for cyber security as well as docs that emphasize more narrow > and > > specific use cases. > > > > Personally, I think I see 3 potential tiers or classifications of docs. > > These are just observations and ideas I had, not necessarily a > prescription > > for organizing docs: > > - Low level tool instructions, eg > > - how do I run the pcap toplogy and then query with the CLI and UI? > > - Platform docs about building on top of Metron, e.g. > > - writing custom parsers, enrichment, and threat Intel (imho we > should > > start to take a more opinionated view of leveraging Stellar as this > > extension point rather than implementing new parser classes in Java) > > - using the profiler for constructing outlier analysis use cases > > - using MAAS for building and deploying models for use in enrichment > > - Docs around more specific use cases that solve specific as opposed to > > more general problems, similar to those we have in the use-cases folder. > > > > I think one of our challenges currently is that our docs could be better > > tailored to the "actors" we've talked about in the past. An individual > SOC > > analyst will have a very different set of interests than would a reseller > > that wants to build on top of our platform to expose new modules and > > functionality to those SOC analyst. And we can, and do, currently support > > both. > > > > > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen wrote: > > > > > Yes, I imagine just a separate top level directory which would contain > > the > > > docs. > > > > > > We would need someone to survey what doc tools are out there and > provide > > > some advice. > > > > > > Maybe we could look around at other open source projects that have done > > > their docs well and emulate them. > > > > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson < > > kylerichards...@gmail.com> > > > wrote: > > > > > > > +1 to separating developer docs and user docs. How should we approach > > > that. > > > > Have a separate doc book? I haven’t had a ton of time to contribute > to > > > code > > > > lately but I’d be happy to help write some of these. > > > > > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen > wrote: > > > > > > > > > Personally, I think the state of our docs and web presence is an > > > > inhibitor > > > > > to growing the Metron community. Unless we can offer concise, > > > compelling > > > > > answers to the basic questions (What can I do with Metron? Who > does > > it > > > > > help? How do I do that?), potential users and contributors are > unable > > > to > > > > > see the value of Metron. > > > > > > > > > > > > > > > > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen > > > wrote: > > > > > > > > > > > I'd like to
Re: [DISCUSS] Getting to a 1.0 release
I agree with that wholeheartedly. The more integrations in and out of the stack at various points, the better. Jon On Mon, Aug 27, 2018, 02:44 Ali Nazemian wrote: > One thing that we could imagine for v1.0 might be an ability to extend > Metron from adding more pipelines to it. For example, being able to extend > Metron to be integrated with other endpoints more easily from Storm > perspective. For example, what if we would like to create other topologies > to write files in ORC directly rather than HDFS or index it to Druid. What > if we would like to build an automatic security response and move to become > SOAR. All these integrations can be done even now and there are other users > that may have done it already. However, providing a clear extension point > can make it easier to contribute other pipelines to the community. I think > by adding this level of extendability, Metron community will grow way > faster by adding more integration that can be available. > > Cheers, > Ali > > On Mon, Aug 20, 2018 at 10:50 PM Casey Stella wrote: > > > I completely agree, Mike. Our docs are either very high level or very > low > > level (and possibly stale) and, worse, aren't aimed at the actors that > > you've stated. > > I think that the HBase project does a good job of providing coherent and > > useable documentation in their "HBase Book" (see > > https://hbase.apache.org/book.html). > > It's not actor-specific, but it is coherent advice for the practical > > practitioner of HBase (both admin and developer) and speaks with one > > voice. I think Metron's need > > is a bit different, but at the minimum some coherent docs that speaks > with > > one voice and has a coherent pitch about what Metron is used for and what > > it isn't used for > > is well needed. > > > > On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Apologies for any spelling mishaps as I'm writing from my phone. > > > > > > I'm for improving our docs. I'd like to see us guide our various > profiles > > > of user towards the specific documentation for the abstraction levels > > > they'll be most interested in working from. I think we should have > > platform > > > docs about how we're a broadly useful, extensible streaming analytics > > > platform for cyber security as well as docs that emphasize more narrow > > and > > > specific use cases. > > > > > > Personally, I think I see 3 potential tiers or classifications of docs. > > > These are just observations and ideas I had, not necessarily a > > prescription > > > for organizing docs: > > > - Low level tool instructions, eg > > > - how do I run the pcap toplogy and then query with the CLI and UI? > > > - Platform docs about building on top of Metron, e.g. > > > - writing custom parsers, enrichment, and threat Intel (imho we > > should > > > start to take a more opinionated view of leveraging Stellar as this > > > extension point rather than implementing new parser classes in Java) > > > - using the profiler for constructing outlier analysis use cases > > > - using MAAS for building and deploying models for use in > enrichment > > > - Docs around more specific use cases that solve specific as opposed to > > > more general problems, similar to those we have in the use-cases > folder. > > > > > > I think one of our challenges currently is that our docs could be > better > > > tailored to the "actors" we've talked about in the past. An individual > > SOC > > > analyst will have a very different set of interests than would a > reseller > > > that wants to build on top of our platform to expose new modules and > > > functionality to those SOC analyst. And we can, and do, currently > support > > > both. > > > > > > > > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen wrote: > > > > > > > Yes, I imagine just a separate top level directory which would > contain > > > the > > > > docs. > > > > > > > > We would need someone to survey what doc tools are out there and > > provide > > > > some advice. > > > > > > > > Maybe we could look around at other open source projects that have > done > > > > their docs well and emulate them. > > > > > > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson < > > > kylerichards...@gmail.com> > > > > wrote: > > > > > > > > > +1 to separating developer docs and user docs. How should we > approach > > > > that. > > > > > Have a separate doc book? I haven’t had a ton of time to contribute > > to > > > > code > > > > > lately but I’d be happy to help write some of these. > > > > > > > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen > > wrote: > > > > > > > > > > > Personally, I think the state of our docs and web presence is an > > > > > inhibitor > > > > > > to growing the Metron community. Unless we can offer concise, > > > > compelling > > > > > > answers to the basic questions (What can I do with Metron? Who > > does > > > it > > > > > > help? How do I do that?), potential users and contributors are > >
Re: [ANNOUNCE] - Apache Metron Slack channel
Invite sent. Jon On Mon, Aug 27, 2018, 02:45 Ali Nazemian wrote: > Can I be invited as well? > > On Thu, Aug 16, 2018 at 4:37 AM Otto Fowler > wrote: > > > Done > > > > > > On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote: > > > > Could I be invited? > > > > On 15-Aug-18 09:48, Michael Miklavcic wrote: > > > + Metron user list > > > > > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > >> Turns out we are able to invite folks on an ad-hoc basis. See > > instructions > > >> here - > > >> > https://cwiki.apache.org/confluence/display/METRON/Community+Resources > > >> > > >> > > >> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic < > > >> michael.miklav...@gmail.com> wrote: > > >> > > >>> It's another option with different features. I imagine many people > will > > >>> use both. > > >>> > > >>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball < > > >>> si...@simonellistonball.com> wrote: > > >>> > > Since this is committers only, would it make more sense to stick to > > IRC? > > Or > > is exclusivity the idea? > > > > On 15 August 2018 at 16:09, Nick Allen wrote: > > > > > Thanks for the instructions! > > > > > > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > >> The Metron community has a Slack channel available for > communication > > >> (similar to the existing IRC channel, only on Slack). > > >> > > >> To join: > > >> > > >> 1. Go to slack.com. > > >> 2. For organization/group, you'll enter "the-asf" > > >> 3. Use your Apache email for your login > > >> 4. Click "Channels" and look for #metron (Created by ottO June 15, > > > 2018) > > >> Best > > >> Mike Miklavcic > > >> > > > > > > -- > > -- > > simon elliston ball > > @sireb > > > > > > > -- > A.Nazemian > -- Jon
Re: Need a slack invite
Invite sent. Jon On Mon, Aug 27, 2018, 03:36 Karthik D B wrote: > Hi Team, > I’m a non-ASF committers, I Would like to join the Metron Slack Channel. > pls. Send an invite. > Thanks, > Karthik DB -- Jon
Need a slack invite
Hi Team, I’m a non-ASF committers, I Would like to join the Metron Slack Channel. pls. Send an invite. Thanks, Karthik DB
Re: [ANNOUNCE] - Apache Metron Slack channel
Can I be invited as well? On Thu, Aug 16, 2018 at 4:37 AM Otto Fowler wrote: > Done > > > On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote: > > Could I be invited? > > On 15-Aug-18 09:48, Michael Miklavcic wrote: > > + Metron user list > > > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> Turns out we are able to invite folks on an ad-hoc basis. See > instructions > >> here - > >> https://cwiki.apache.org/confluence/display/METRON/Community+Resources > >> > >> > >> On Wed, Aug 15, 2018 at 9:23 AM Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> > >>> It's another option with different features. I imagine many people will > >>> use both. > >>> > >>> On Wed, Aug 15, 2018, 9:14 AM Simon Elliston Ball < > >>> si...@simonellistonball.com> wrote: > >>> > Since this is committers only, would it make more sense to stick to > IRC? > Or > is exclusivity the idea? > > On 15 August 2018 at 16:09, Nick Allen wrote: > > > Thanks for the instructions! > > > > On Wed, Aug 15, 2018 at 10:22 AM, Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > >> The Metron community has a Slack channel available for communication > >> (similar to the existing IRC channel, only on Slack). > >> > >> To join: > >> > >> 1. Go to slack.com. > >> 2. For organization/group, you'll enter "the-asf" > >> 3. Use your Apache email for your login > >> 4. Click "Channels" and look for #metron (Created by ottO June 15, > > 2018) > >> Best > >> Mike Miklavcic > >> > > > -- > -- > simon elliston ball > @sireb > > -- A.Nazemian
Re: [DISCUSS] Getting to a 1.0 release
One thing that we could imagine for v1.0 might be an ability to extend Metron from adding more pipelines to it. For example, being able to extend Metron to be integrated with other endpoints more easily from Storm perspective. For example, what if we would like to create other topologies to write files in ORC directly rather than HDFS or index it to Druid. What if we would like to build an automatic security response and move to become SOAR. All these integrations can be done even now and there are other users that may have done it already. However, providing a clear extension point can make it easier to contribute other pipelines to the community. I think by adding this level of extendability, Metron community will grow way faster by adding more integration that can be available. Cheers, Ali On Mon, Aug 20, 2018 at 10:50 PM Casey Stella wrote: > I completely agree, Mike. Our docs are either very high level or very low > level (and possibly stale) and, worse, aren't aimed at the actors that > you've stated. > I think that the HBase project does a good job of providing coherent and > useable documentation in their "HBase Book" (see > https://hbase.apache.org/book.html). > It's not actor-specific, but it is coherent advice for the practical > practitioner of HBase (both admin and developer) and speaks with one > voice. I think Metron's need > is a bit different, but at the minimum some coherent docs that speaks with > one voice and has a coherent pitch about what Metron is used for and what > it isn't used for > is well needed. > > On Sat, Aug 18, 2018 at 1:00 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Apologies for any spelling mishaps as I'm writing from my phone. > > > > I'm for improving our docs. I'd like to see us guide our various profiles > > of user towards the specific documentation for the abstraction levels > > they'll be most interested in working from. I think we should have > platform > > docs about how we're a broadly useful, extensible streaming analytics > > platform for cyber security as well as docs that emphasize more narrow > and > > specific use cases. > > > > Personally, I think I see 3 potential tiers or classifications of docs. > > These are just observations and ideas I had, not necessarily a > prescription > > for organizing docs: > > - Low level tool instructions, eg > > - how do I run the pcap toplogy and then query with the CLI and UI? > > - Platform docs about building on top of Metron, e.g. > > - writing custom parsers, enrichment, and threat Intel (imho we > should > > start to take a more opinionated view of leveraging Stellar as this > > extension point rather than implementing new parser classes in Java) > > - using the profiler for constructing outlier analysis use cases > > - using MAAS for building and deploying models for use in enrichment > > - Docs around more specific use cases that solve specific as opposed to > > more general problems, similar to those we have in the use-cases folder. > > > > I think one of our challenges currently is that our docs could be better > > tailored to the "actors" we've talked about in the past. An individual > SOC > > analyst will have a very different set of interests than would a reseller > > that wants to build on top of our platform to expose new modules and > > functionality to those SOC analyst. And we can, and do, currently support > > both. > > > > > > On Sat, Aug 18, 2018, 9:34 AM Nick Allen wrote: > > > > > Yes, I imagine just a separate top level directory which would contain > > the > > > docs. > > > > > > We would need someone to survey what doc tools are out there and > provide > > > some advice. > > > > > > Maybe we could look around at other open source projects that have done > > > their docs well and emulate them. > > > > > > On Sat, Aug 18, 2018, 10:57 AM Kyle Richardson < > > kylerichards...@gmail.com> > > > wrote: > > > > > > > +1 to separating developer docs and user docs. How should we approach > > > that. > > > > Have a separate doc book? I haven’t had a ton of time to contribute > to > > > code > > > > lately but I’d be happy to help write some of these. > > > > > > > > On Sat, Aug 18, 2018 at 9:48 AM Nick Allen > wrote: > > > > > > > > > Personally, I think the state of our docs and web presence is an > > > > inhibitor > > > > > to growing the Metron community. Unless we can offer concise, > > > compelling > > > > > answers to the basic questions (What can I do with Metron? Who > does > > it > > > > > help? How do I do that?), potential users and contributors are > unable > > > to > > > > > see the value of Metron. > > > > > > > > > > > > > > > > > > > > On Sat, Aug 18, 2018 at 9:42 AM, Nick Allen > > > wrote: > > > > > > > > > > > I'd like to see us focus on improving our docs before a version > > 1.0. > > > > > > Right now we just stitch together a bunch of READMEs, which is a > > > great > > > > > > stride from where we started, but is not ideal. > > > >
