Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

*/ > 198 private static void addTestFieldMappings(JSONObject template, String > docType) { > 199 Map mappings = (Map) template.get("mappings"); > 200 Map docTypeJSON = (Map) mappings.get(docType); > 201 Map properties = (Map) docTypeJSON.get("properties&q

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

Can you diff the trees to be sure? On November 15, 2018 at 17:52:40, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: So amazingly, this still has results in conflicts, but I am able to resolve them manually in a sensible fashion. git merge -X theirs es-rebased CONFLICT (rename/rename): R

Re: [DISCUSS] Attribution and merging the Elasticsearch client migration

Proper attribution and the correct code are the most important things, not the number of commits. On November 15, 2018 at 16:29:04, Justin Leet (justinjl...@gmail.com) wrote: I took a look at this with Mike a bit, and it seems like it's pretty painful and without a clear way to avoid remerging c

New WIP PR, a new Full Dev option

I have submitted a WIP PR [#1261](https://github.com/apache/metron/pull/1261) that makes it possible to run / try the metron full dev environment with only Vagrant, VirtualBox and Docker installed, as opposed to having to have all the dev tools and ansible at the right version. I think this would

Re: [DISCUSS] Slack Channel Use

up 100% and the dev list traffic is flat >>> as >>> compared to last quarter. That's not to say that we couldn't stand more >>> discussion on the lists, but a lot of the dev discussion happens on >> github >>> and JIRA and I'm happy to see

Broken build at the moment

We have a stellar test for date format that is broken because of the daylight savings change. Justin and I have been working through it and I’ll have a PR as soon as my travis build completes. https://issues.apache.org/jira/browse/METRON-1864 Just a heads up that any new builds ( at least in the

Re: [DISCUSS] Day 1 User Experience - Getting Metron Running

What is the metron on docker part? On October 26, 2018 at 14:37:48, Nick Allen (n...@nickallen.org) wrote: > Yeah I would +1 katakoda. Has anyone used or have a history with KataKoda? I'd hate to invest time in a hosted solution if the provider isn't going to be around. That's a definite 'con'

metron-elasticsearch integration tests failing after merging in master

https://travis-ci.org/ottobackwards/metron/jobs/445723343 Anyone having ES test problems? Anyone shed any light on this.

Re: [DISCUSS] Slack Channel Use

es? I have never used a search engine and > > > uncovered the answer to my problem in a Slack archive. > > > > > > On Mon, Oct 22, 2018 at 5:05 PM Otto Fowler > > > wrote: > > > > > > > According to Greg Stein, an infra ad

Re: Invite to Slack Channel

t; > > > > > > > > > On Wed, Oct 17, 2018 at 7:33 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> wrote: > > > > > > > > > Sent > > > > > On Wed, Oct 17, 2018 at 7:23 AM Tibor Meller <

Re: [DISCUSS] Slack Channel Use

According to Greg Stein, an infra admin on the NiFi slack, the ASF slack that metron is in IS the standard plan, not the free one and is searchable past 10,000 messages. On October 22, 2018 at 15:35:51, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: ...From an archival and broader reac

Re: [DISCUSS] Slack Channel Use

These questions also occurred on the IRC channel. The difference is that there are more than Jon and I answering now. On October 22, 2018 at 12:18:08, Nick Allen (n...@nickallen.org) wrote: It seems that we are seeing a lot of Metron usage and support questions on the Slack Channel. These are q

Re: [DISCUSS] Stellar REST client

I believe the issue of introducing and supporting higher latency enrichments is a systemic one, and should be solved as such, with the rest and other higher latency enrichments build on top of that framework. On October 19, 2018 at 12:22:28, Ryan Merriman (merrim...@gmail.com) wrote: Thanks Ca

Re: Bro plugin unit tests failing

It is INFRA, see INFRA-17091 for example. On October 12, 2018 at 20:47:24, zeo...@gmail.com (zeo...@gmail.com) wrote: So it seems that the last commit before the 0.2 release of metron-bro-plugin-kafka broke the one basic unit test that we had. Since metron 0.6.0 pins to 0.1 < https://github.com/

Re: Custom parser using Jackson instead of json-simple

The ParserBolt is written to JSON simple, so although the interface is in practice it is . The answer is no right now. Feel free to open a jira. On October 5, 2018 at 02:52:37, Muhammed Irshad (irshadkt@gmail.com) wrote: Hi All, Is it not possible to use any Json library other than json-

Re: Invite to Slack Channel

Done On October 4, 2018 at 05:35:06, Tamás Fodor (ftamas.m...@gmail.com) wrote: Hello, Michael, can you add me as well? Thank you in advance! Tamas On Wed, Oct 3, 2018 at 4:27 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Sent > > On Wed, Oct 3, 2018 at 8:17 AM Shane Ardell

Re: [DISCUSS] Feature Branch guidance

This is all well and good for feature branches, but does nothing for Simon and the type of work he attempted. If we agree that features do not have architectural changes, then we also need to codify how we handle that level of change, assuming anyone is optimistic enough to attempt such a thing in

Re: Metron dev environments moving to require Ansible 2.4+

;? It was the only reference I could find on the wiki. All of the READMEs should be updated as a part of the PR, but feel free to provide your input if I missed anything. Jon On Fri, Sep 28, 2018 at 10:15 AM Otto Fowler wrote: > We should make sure the non-source documentation is updated

Re: Metron dev environments moving to require Ansible 2.4+

We should make sure the non-source documentation is updated On September 28, 2018 at 09:32:52, zeo...@gmail.com (zeo...@gmail.com) wrote: Hi All, As it currently sits, once METRON-1758 is merged into the code base, Ansible 2.4 or later will be requir

Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated

viewed and in reviewing PRs. Could you share your experience there? On Wed, Sep 12, 2018 at 1:36 PM Otto Fowler wrote: > Are you referring to the dependencies check against the csv? > > > On September 12, 2018 at 13:09:48, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: &

Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated

Are you referring to the dependencies check against the csv? On September 12, 2018 at 13:09:48, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: I'm not sure I fully understand what is out of date. I know I have personally modified our licenses a couple times in the past and used an autom

Re: [DISCUSS] Feature branches post-merge

I would drop them. I’ve already clean up FB’s around dead things. On September 6, 2018 at 13:42:55, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: What are we doing with feature branches once they're complete and merged into master? Is our expectation that we'll keep feature branches i

Re: IRC Channel -> OPS?

Damn, I was hoping not. It will never happen now On August 29, 2018 at 15:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote: Isn't it Casey? Jon On Wed, Aug 29, 2018, 08:41 Otto Fowler wrote: > Who has ops in the irc channel? > Can you pop in and set the topic to something like:

Re: [DISCUSS] Contributing a General Purpose Regex Parser

I would like to see a PR on this. Do you have an example of a second type of log where this would be useful? Besides something syslog-y? There is a PR out for a Syslog RFC 5424 parser that handles that ( including structured data, which I don’t know if you have in your parser ). What may be more

IRC Channel -> OPS?

Who has ops in the irc channel? Can you pop in and set the topic to something like: “There is an ASF slack with an active metron channel, please email dev@metron.apache.org and request an invite”

Re: [DISCUSS] Getting to a 1.0 release

gt; > > > > > > The docs should be separate from the code base to allow for an > > > > > > organization that is focused on the user rather than the > > > > implementation. > > > > > > This allows the READMEs to focus on the devel

package.lock changes during build?

I just did a PR, can saw that the package.lock file for alerts-ui was changed, with updated versions. I did *not* change the file, nor anything in metron-interface. That seems to imply that this file is changed or updated by something that happens during building or deploying full dev. Is this tru

Re: [DISCUSS] Pcap query branch completion

gt; - Documentation/blueprint for YARN configuration > > You make a good point. A YARN tuning guide for Metron does sound useful. > I will add a follow on Jira. > > On Mon, Aug 13, 2018 at 4:53 PM, Otto Fowler > wrote: > >> >> - Date range limits on queries >>

Re: [DISCUSS] Getting to a 1.0 release

service api to get at it. I’m all for that too, but think it needs more thought than the ticket captures. Simon On 15 Aug 2018, at 20:53, Otto Fowler wrote: https://issues.apache.org/jira/browse/METRON-343 On August 15, 2018 at 15:47:24, Simon Elliston Ball ( si...@simonellistonball.com) wrote

Re: [DISCUSS] Getting to a 1.0 release

https://issues.apache.org/jira/browse/METRON-106 At least making sure it is met and closing it On August 15, 2018 at 15:53:02, Otto Fowler (ottobackwa...@gmail.com) wrote: https://issues.apache.org/jira/browse/METRON-343 On August 15, 2018 at 15:47:24, Simon Elliston Ball ( si

Re: [DISCUSS] Getting to a 1.0 release

doing to follow up on the Knox Feature was to add Ranger integration for securing and auditing configs, and potentially extending to the index destinations. Do you think that would cover the secure storage concept? Simon > On 15 Aug 2018, at 20:39, Otto Fowler wrote: > > Secure storage of

Re: [DISCUSS] Getting to a 1.0 release

Secure storage off the top of my head On August 15, 2018 at 14:49:26, zeo...@gmail.com (zeo...@gmail.com) wrote: So, as has been discussed in a few < https://lists.apache.org/thread.html/0445cd8f94dfb844cd5a23ac3eeca04c9f44c9d8f269c6ef12cb3598@%3Cdev.metron.apache.org%3E> other < https://lists.a

Re: [ANNOUNCE] - Apache Metron Slack channel

Done On August 15, 2018 at 14:22:45, Vets, Laurens (laur...@daemon.be) wrote: Could I be invited? On 15-Aug-18 09:48, Michael Miklavcic wrote: > + Metron user list > > On Wed, Aug 15, 2018 at 10:38 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> Turns out we are able to invite

Re: [DISCUSS] Metron Parsers in Nifi

you > don't emit any records at all for a flow file, it errors, which is not > strictly speaking an error, but yeah, we can certainly control things like > filtering errors aside from this. I would say this was a nifi bug > (debatably) which should be fixed on that side. > > Si

Re: [DISCUSS] Pcap query branch completion

- Date range limits on queries I took the point the wrong way apparently, sorry, I withdraw. I thought you meant allow specifying a limit on the query, not the system imposing a limit. This should be documented with a warning or something - UI should manage a queue/history of jobs I was thinkin

Re: [DISCUSS] Metron Parsers in Nifi

/java/org/apache/nifi/syslog/Syslog5424Reader.java On August 13, 2018 at 09:26:50, Otto Fowler (ottobackwa...@gmail.com) wrote: If we can do the record readers ourselves ( with the parsers inside them ) we can handle the returns. I’ll be doing the net flow 5 readers once the net flow 5 processor

Re: [DISCUSS] Metron Parsers in Nifi

pain out of setup for sensors). Simon On 9 August 2018 at 16:42, Otto Fowler wrote: > I would say that > > - For each configuration parameter we want to pull in, it should be > explicitly configured through a property as well as through a controller > service that accesses th

Re: [DISCUSS] Pcap query branch completion

- Job cleanup/TTL Documented at least, or a helper script to help yourself if you are in a situation - Expose the Query filter (vs Fixed) in the UI Follow on - Date range limits on queries I don’t see how this won’t be immediately required. I would do this for minimum viable. - Pcap query

Re: [DISCUSS] Metron Parsers in Nifi

run our set of parsers, lets users build new parsers (and don't block specialized NiFi implementations that exploit NiFi's feature set), and lets us get things configured in a relatively consistent manner, without losing features, and hopefully requiring a pretty minimal slice of Metron to

Re: [DISCUSS] Metron Parsers in Nifi

ogies. The integration would be that the > > NiFi processor parses the data and pushes it straight into the enrichment > > topic, saving us the resources of having multiple parsers in storm > > > > Thanks, > > James > > > > 07.08.2018, 11:29, &quo

Re: [DISCUSS] Metron Parsers in Nifi

able back then. > On Wed, Aug 8, 2018 at 11:46 PM Otto Fowler wrote: > >> I’m seeing >> >> https://github.com/apache/nifi/blob/master/nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/RecordReader.java#L34 >> being quoted as a reason to NOT build

article on swagger and ambari

Worth checking out. https://community.hortonworks.com/articles/210091/how-to-use-swagger-with-ambari-explore-ambari-rest.html

Re: [DISCUSS] Metron Parsers in Nifi

should be a parallel improvement, not a conflicting one. On Tue, Aug 7, 2018 at 11:50 AM Otto Fowler wrote: > A Metron Processor itself isn’t really necessary. A MetronRecordReader ( > either the megalithic or a reader per format ) would be a good approach. > Then have StellarTransformRecor

Re: [DISCUSS] Metron Parsers in Nifi

configure the Processor itself with the data (just don't set up a controller service and provide the json or whatever as one of our properties). On Tue, Aug 7, 2018 at 10:12 AM Otto Fowler wrote: > I think this is a good idea. As I mentioned in the other thread I’ve been > doing a lot

Re: [DISCUSS] Metron Parsers in Nifi

I think this is a good idea. As I mentioned in the other thread I’ve been doing a lot of work on Nifi recently. I think the important thing is that what is done should be done the NiFi way, not bolting the Metron composition onto Nifi. Think of it like the Tao of Unix, the parsers and components

Re: [DISCUSS] Batch Profiler

I think the feature branch is a good idea, but what is in the feature branch or feature branches will have to shake out. I agree in concept with what you have in the jira, but I have two points. 1. We will need a break down of introducing Spark to the stack - required version due to HDP

Re: Metron docker compose fails

metron docker is in the /contrib area because it deviates from the main or ‘official’ development code. Simply put, the docker stuff doesn’t use ambari, and can lag behind in implementation changes because it all custom. If it isn’t working, then you can and should log a jira on it. There have be

Re: Security Feature Branch?

full dev, hence the one PR one unit approach. Does that work, or do we want to review on the basis of a series of untestable bits, and then a final working build PR that pulls it together? Simon On 12 July 2018 at 16:00, Otto Fowler wrote: > Our policy in the past on such things is to requ

Re: Security Feature Branch?

A discussion thread on what you have come up with, the choices you made would be warranted as well. On July 12, 2018 at 11:00:47, Otto Fowler (ottobackwa...@gmail.com) wrote: Our policy in the past on such things is to require that they are broken into small reviewable chunks on a feature

Re: Security Feature Branch?

Our policy in the past on such things is to require that they are broken into small reviewable chunks on a feature branch, even if the end to end working version was more ‘usable’. On July 12, 2018 at 10:51:30, Simon Elliston Ball ( si...@simonellistonball.com) wrote: I've been doing some work

Re: Performance comparison between Grok and Java regex

:19, Muhammed Irshad (irshadkt@gmail.com) wrote: Otto Fowler, Thanks for the reply. I saw it uses same Java regex under the hood. I got bit sceptic by seeing this open issue <https://github.com/thekrakken/java-grok/issues/75> in java-grok which says grok is much slower when compared wit

Re: Performance comparison between Grok and Java regex

Java-Grok IS java regex. It is just a DSL over Java regex. It takes grok expressions ( that can reference other expressions and be compound ) and parses/resolves them and then builds one big regex out of them. Also, Groks, once parsed / used are re-used, so at that point they are like compiled re

Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master

ed if we get a couple +1's on the PR, it's > essentially voting anyway, but this is pretty new in terms of process. > > > > On Fri, Jun 22, 2018 at 12:53 PM Otto Fowler > wrote: > >> If all the PR’s are on master->feature branch. Why do we need testing? >

Re: [DISCUSS] Merging Solr feature branch (METRON-1416) into master

If all the PR’s are on master->feature branch. Why do we need testing? this is almost a vote situation. On June 22, 2018 at 12:01:11, Justin Leet (justinjl...@gmail.com) wrote: The (formerly) active PRs are now merged in and closed. We don't seem to have defined way to merge a feature branch

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

-jiras- On June 13, 2018 at 10:30:26, Simon Elliston Ball ( si...@simonellistonball.com) wrote: That’s where something like the Nifi solution would come in... With the PutEnrichment processor and a ProcessHttpRequest processor, you do have a web service for loading enrichments. We could probab

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

Do we even have a jira? If not maybe Carolyn et. al. can write one up that lays out some requirements and context. On June 13, 2018 at 10:04:27, Casey Stella (ceste...@gmail.com) wrote: no, sadly we do not. On Wed, Jun 13, 2018 at 10:01 AM Carolyn Duby wrote: > Agreed….Streaming enrichments

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

) wrote: I agree with Simon here, the benefit of providing NiFi tooling is to enable NiFi to use our infrastructure (e.g. our parsers, MaaS, stellar enrichments, etc). This would tie it to Metron pretty closely. On Tue, Jun 5, 2018 at 3:12 PM Otto Fowler wrote: > Nifi releases more often t

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

of course that does increase our release and test burden. On 5 June 2018 at 10:55, Otto Fowler wrote: > Similar to Bro, we may need to release out of cycle. > > > > On June 5, 2018 at 13:17:55, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > Do you mean

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

Similar to Bro, we may need to release out of cycle. On June 5, 2018 at 13:17:55, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Do you mean in the sense of a separate module, or are you suggesting we go as far as a sub-project? On 5 June 2018 at 10:08, Otto Fowler wrote: > If

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

ar to all FlowFile attributes outputting the resulting stellar variable space to either attributes or as json in the content. Is it worth us creating an nifi-metron-bundle. Happy to kick that off, since I'm half way there. Simon On 5 June 2018 at 08:41, Otto Fowler wrote: > We hav

Re: [DISCUSS] Field conversions

(eol) will need to not the bullet with ES compatibility as some point. Simon > On 5 Jun 2018, at 17:17, Otto Fowler wrote: > > Are there consequences with Kibana as well? queries, visualizations, > templates they may have? > > > On June 5, 2018 at 12:03:44, Nick Allen (n...

Re: [DISCUSS] Field conversions

Are there consequences with Kibana as well? queries, visualizations, templates they may have? On June 5, 2018 at 12:03:44, Nick Allen (n...@nickallen.org) wrote: I just don't know if telling users to do a bulk upgrade of their indices is sufficient enough of an upgrade path. I would expect some

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

We have jiras about ‘diverting’ and reading from nifi flows already On June 5, 2018 at 11:11:45, Casey Stella (ceste...@gmail.com) wrote: I'd be in strong support of that, Simon. I think we should have some other NiFi components in Metron to enable users to interact with our infrastructure from

Re: Writing enrichment data directly from NiFi with PutHBaseJSON

PutMetronEnrichementRecords* ;) On June 5, 2018 at 10:32:43, Simon Elliston Ball ( si...@simonellistonball.com) wrote: Do we, the community, think it would be a good idea to create a PutMetronEnrichment NiFi processor for this use case? It seems a number of people want to use NiFi to manage and

Re: [DISCUSS] Field conversions

It is still our user list and dev list that will have the burden of talking folks through that. On June 5, 2018 at 09:58:32, Casey Stella (ceste...@gmail.com) wrote: To be clear, I'm not even suggesting that we create any tooling here. I'd say just a reference to the ES docs and a call-out in Up

Re: [VOTE] Metron Release Candidate 0.5.0-RC2

+1 binding Ran the script Validated ui + data in ambari/storm/config ui On May 31, 2018 at 14:35:20, Justin Leet (justinjl...@gmail.com) wrote: This includes a couple fixes from master, in particular two issues that were problematic, METRON-1586

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

in Leet >> wrote: >> >> > I'm going to go ahead and cancel RC1, since METRON-1544 looks pretty >> set. >> > >> > A new release candidate will be cut. >> > >> > Results (including my own vote): >> > +1 >> > Nick A

Re: [DISCUSS] Refactoring

d I'd suggest we clarify to be cosmetic refactoring solely due to >> readability concerns. >> >> Just my $0.02 >> >> On Tue, May 29, 2018 at 7:40 PM Otto Fowler >> wrote: >> >>> On top of this, refactoring under another PR’s goals tends to

Re: [DISCUSS] Refactoring

On top of this, refactoring under another PR’s goals tends to be less documented as to the intent and effect. +1 for the idea, we should have a vote round or edit round on the doc’s specific text. Although I will say, that some things it doesn’t matter how much you break them up wrt reviews. We sh

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

se since there were two new commits, but I > don't think it was included in this round. > > Jon > > On Sat, May 26, 2018, 10:22 Otto Fowler wrote: > > > Is there a BRO RC # for this? > > > > > > On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.o

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

exit status from /var/tmp/rpm-tmp.BzHlk8 (%install) On May 27, 2018 at 10:53:49, Otto Fowler (ottobackwa...@gmail.com) wrote: Failed tests: CachingStellarProcessorTest.testCaching:73 expected:<6> but was:<5> I thought we landed a fix for this? On May 27, 2018 at 08:24:19, zeo...

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

t was included in this round. Jon On Sat, May 26, 2018, 10:22 Otto Fowler wrote: > Is there a BRO RC # for this? > > > On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org) wrote: > > +1 Release this package as Apache Metron 0.5.0-RC1 > > Ran through all validation

Re: [VOTE] Metron Release Candidate 0.5.0-RC1

Is there a BRO RC # for this? On May 25, 2018 at 14:53:25, Nick Allen (n...@nickallen.org) wrote: +1 Release this package as Apache Metron 0.5.0-RC1 Ran through all validation steps using the `metron-rc-check` script, which included running all the tests, license checks, and spun-up the CentOS

Re: [DISCUSS] parser ES + Solr schema abstraction

process would be config change triggering schema inference triggering diff to old schema optionally triggering a net new version. Does they make sense? Simon On 22 May 2018, at 19:33, Otto Fowler wrote: I’ve also talked with J. Zeolla conceptually storing data in hdfs relative to the version of the

Re: [DISCUSS] parser ES + Solr schema abstraction

fields) but not to others (removing or reordering fields). This can be resolved by sensible versioning and history aware schema generation. Simon On 22 May 2018 at 15:23, Otto Fowler wrote: > Yes Simon, when I say ‘whatever we would call the complete parse/enrich > path’ that is what

Re: [DISCUSS] parser ES + Solr schema abstraction

on to generate all the required artefacts for whatever storage it ends up in. Essentially, composable partial schema units from each component, which add up at the end. Does that make sense? Simon On 22 May 2018 at 14:10, Otto Fowler wrote: > We have discussed in the past as part of 777 (

[DISCUSS] parser ES + Solr schema abstraction

We have discussed in the past as part of 777 ( moment of silence…. ) the idea that parsers/sensors ( or whatever we would call the complete parse/enrich path ) could define a their ES or Solr schemas so that they can be ‘installed’ as part of metron and remove the requirement for a separate install

Re: Request for Comment on new Syslog 5424 Parsing library

I am open to adding new syslog parsers or parser ‘specifications’ as I have termed them in. Possibly using grok in the background. On May 21, 2018 at 07:03:40, Otto Fowler (ottobackwa...@gmail.com) wrote: Thanks Ahmed. At the moment, I’m only concerned with RFC 5424 formatted syslog <ht

Re: Request for Comment on new Syslog 5424 Parsing library

/tiki/lce/index.php> From: Casey Stella Sent: May 18, 2018 10:59 AM To: dev@metron.apache.org Subject: Re: Request for Comment on new Syslog 5424 Parsing library Cool! I'd welcome a syslog parser! On Fri, May 18, 2018 at 10:02 AM Otto Fowler wrote: > There

Request for Comment on new Syslog 5424 Parsing library

There have been some issues and talk about they way we parse syslog, and the deficiencies of our grok and regex based approaches, mainly not supporting structured data as I recall. I played around with it some and decided to try to write an Antlr grammar based on the RFC 5424 spec BNF to parse vali

Re: [DISCUSS] Metron release 0.5.0

d, May 16, 2018, 7:01 AM Otto Fowler wrote: > > > My question is: Is updating the version a .4->.5 worthy change or would > > adding Solr be that change? > > Should we do another, last .4.x release and bump to .5 when solr hits? > > > > > > On May 15, 2018

Re: [DISCUSS] Metron release 0.5.0

My question is: Is updating the version a .4->.5 worthy change or would adding Solr be that change? Should we do another, last .4.x release and bump to .5 when solr hits? On May 15, 2018 at 17:31:27, Nick Allen (n...@nickallen.org) wrote: +1 That plan works for me. IMHO, I don't think there ar

Re: [DISCUSS] Pcap panel architecture

fairly tractable. > > FYI, I've been doing a lot of thinking around data security, API and > configuration security and auditing recently, but I suspect that is a > different discuss thread. I'll kick something off shortly with a few > thoughts. > > I see a lot of th

Re: [DISCUSS] Pcap panel architecture

t; > > > to > > > > > do > > > > > > > is generate some pcap data first. > > > > > > > > > > > > > > On Tue, May 8, 2018 at 4:17 PM, Michael Miklavcic < > > > > > > > michae

Re: [DISCUSS] Release?

gt; >> 3 months ago METRON-1413 Add Metron Commit Tool (nickwallen) >> > closes >> > > > > >> apache/metron#902 >> > > > > >> 3 months ago METRON-1429 SearchIntegrationTest refactor >> > (merrimanr) >> > > > >

Re: [DISCUSS] Release?

Can you run the issues included script and post that for us to see? On May 9, 2018 at 11:14:11, Casey Stella (ceste...@gmail.com) wrote: Is it about time for a release? I know we got some substantial performance changes in since the last release. I think we might have a justification for a relea

Re: [DISCUSS] Pcap panel architecture

> setting "yarn.timeline-service.enabled" in Ambari to false and then I > > get > > this error: > > > > Unable to parse > > '/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a > > URI, check the setting for mapreduce.application

Re: [DISCUSS] Pcap panel architecture

client to that service, the specializes the service operation for the way we want pcap to work. We can then re-use the generic service for other long running yarn things….. On May 7, 2018 at 09:56:51, Otto Fowler (ottobackwa...@gmail.com) wrote: RE: Tracking v. users The submittal and tracking can

Re: [DISCUSS] Pcap panel architecture

dependencies as a special module? That seems like a very attractive option to me. On Fri, May 4, 2018 at 8:39 AM, Otto Fowler wrote: > From my response on the other thread, but applicable to the backend stuff: > > "The PCAP Query seems more like PCAP Report to me. You are generatin

Re: [DISCUSS] Pcap UI user requirements

investigate some case where the user want to see the whole packet (all the bits and bytes). Like in wireshark, something interactive no? 2018-05-04 14:33 GMT+01:00 Otto Fowler : > The PCAP Query seems more like PCAP Report to me. You are generating a > report based on parameters. > That

Re: [DISCUSS] Pcap panel architecture

>From my response on the other thread, but applicable to the backend stuff: "The PCAP Query seems more like PCAP Report to me. You are generating a report based on parameters. That report is something that takes some time and external process to generate… ie you have to wait for it. I can almost

Re: [DISCUSS] Pcap UI user requirements

The PCAP Query seems more like PCAP Report to me. You are generating a report based on parameters. That report is something that takes some time and external process to generate… ie you have to wait for it. I can almost imagine a flow where you: * Are in the AlertUI * Ask to generate a PCAP repo

Re: [DISCUSS] Pcap panel architecture

ong position on this other than 1) management is a different feature set from drilling into threat intel, yet many apps still have their management UI combined with the end user experience and 2) we should probably consider pcap in context of a workflow with alerts. On Thu, May 3, 2018 at 4:19 PM

Re: [DISCUSS] Pcap panel architecture

eparate service or call out to the pcap_query.sh script from our existing REST app. I could go either way really. I'm just not excited about all the MPack code we have to write for a new component. Maybe it won't be that bad. On Thu, May 3, 2018 at 2:50 PM, Otto Fowler wrote: > Fi

Re: [DISCUSS] Pcap panel architecture

First thought is why the Alerts-UI and Not a dedicated Query UI? On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) wrote: We are planning on adding the pcap query feature to the Alerts UI. Before we start this work, I think it is important to get community buy in on the architectu

Re: [VOTE] Development Guidelines Addendum on Inactive Pull Requests

+1 On April 20, 2018 at 09:30:30, Nick Allen (n...@nickallen.org) wrote: I am proposing the following addition to the project's development guidelines [1]. Based on these guidelines, an abandoned pull request can be closed in roughly 6 weeks time (4 weeks of inactivity plus 2 weeks to respond to

Re: [DISCUSS] Metron RPM spec changelog

The other approach would mean just > > doing a git log on the spec file and grabbing the delta since last > release. > > Side note, I kind of like the idea of having the Jira ticket number in > the > > comment like that in the second example. What do you guys think? > >

Re: [DISCUSS] Metron RPM spec changelog

r approach would mean just > doing a git log on the spec file and grabbing the delta since last release. > Side note, I kind of like the idea of having the Jira ticket number in the > comment like that in the second example. What do you guys think? > > Mike > > > On Wed, A

Re: [DISCUSS] Metron RPM spec changelog

I think having the spec file updated with the changes per release is fine, but is the release manager going to do that? If so then the docs need to be updated. Also, we *should* true up any missing entries from the file now. On April 18, 2018 at 11:02:35, Casey Stella (ceste...@gmail.com) wrot

Re: [DISCUSS] Inactive PRs

thing, instead of "submitter", I'll stick with "contributor" because I use that everywhere else. A pull request is 'inactive' if no comments or updates have been made by the contributor in the previous 6 weeks. On Fri, Apr 13, 2018 at 3:06 PM, Otto Fowler wrot

<    1   2   3   4   5   6   7   >