Newbie

Hi All, I've recently joined both the dev and user mailing lists for Metron and thought I'd give a short intro before I started in with questions. I've been working in the Cyber Security space for a little over 4 years now primarily as a SIEM Engineer, SOC Analyst, and SOC Manager. In my current

Re: Newbie

est, > > Casey > > On Tue, Jul 26, 2016 at 9:25 PM, Kyle Richardson < > kylerichards...@gmail.com> > wrote: > > > Hi All, > > > > I've recently joined both the dev and user mailing lists for Metron and > > thought I'd give a short intro befor

Re: [DISCUSS] Contributions with multiple authors and requisite modifications to the development guide

+1 I think this provides a reasonable set of expectations for the author and committer. -Kyle On Tue, Jan 31, 2017 at 10:46 AM, Nick Allen wrote: > I am in agreement with what I am reading. Something along the lines of the > following. > >- Best effort should be made

Re: [Discuss] Direction of metron-docker

I like the idea of porting some of the integration tests to metron-docker. I believe the maven plugin used in the rpm-docker project could be used to support that goal. I agree with Ryan in that I see this as more of a toolbox for developers than a supported deployment method. That is the vain I

Re: [DISCUSS] Metron Management UI

This looks awesome! A great starting point. Like Jon, I'm looking forward to kicking the tires. -Kyle On Wed, Feb 1, 2017 at 10:18 AM, Ryan Merriman wrote: > Jon, I've done my best to keep this UI in sync with the API PR so you can > play around with it now if you want.

Re: [Discuss] Direction of metron-docker

My working assumption has been that metron-docker would provide a lightweight development environment without all of the overhead of a full cluster. The ability to quickly spin up and down specific components while coding is a big win in my book. Unfortunately, since metron-docker containers

Re: [DISCUSS] Coding style via checkstyle

+1 to longer line lengths and blanket reformatting. Personally, I see IDE integration as a must for adoption of checkstyle. -Kyle On Wed, Feb 22, 2017 at 1:39 AM, Matt Foley wrote: > +1, so do I. Also like the idea of providing the necessary IntelliJ > specification. > > On

Re: [DISCUSS] Management of Elastic and other index schemas

I personally like the idea of a typed schema per parser that we could translate to multiple targets. This would allow us a lot more modularity and extensibility in indexing down the road. -Kyle On Fri, Feb 17, 2017 at 1:59 PM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > That

Re: new committer: Josh Meyer

ber enables assistance with the management > and to guide the direction of the project. > > > 23.02.2017, 13:14, "James Sirota" <jsir...@apache.org>: > > The Podling Project Management Committee (PPMC) for Apache Metron > (Incubating) > > has asked Kyle Richar

Odd integration-test failures on Fedora/CentOS for RC5

Ok, I've tried this so many times I may be going crazy, so thought I'd ask the community for a sanity check. I'm trying to verify RC5 and I keep running into the same integration test failures but only on my Fedora (24 and 25) and CentOS 7 systems. It passes fine on my Macbook. It always fails

Re: Odd integration-test failures on Fedora/CentOS for RC5

filesystem specific? Apologies if I'm just being dense but I'd really like to understand why this consistently fails on some platforms and not others. -Kyle > On Feb 25, 2017, at 9:07 AM, Kyle Richardson <kylerichards...@gmail.com> > wrote: > > Ok, I've tried this so many ti

Re: custom date format required for snort, but not working

You're correct, a ZonedDateTime requires a year. I ran into this when parsing the RFC3164 syslog timestamps. Glad he was able to find the config option to enable the year in Snort. -Kyle On Tue, Feb 21, 2017 at 7:59 AM, Otto Fowler wrote: > ok - > > # Configure Snort

Re: [DISCUSS] Moving GeoIP management away from MySQL

+1 Agree with David's order -Kyle On Mon, Jan 16, 2017 at 12:41 PM, David Lyle wrote: > Def agree on the parity point. > > I'm a little worried about Supervisor relocations for non-HBase solutions, > but having much of the work done for us by MaxMind changes my preference

Re: [VOTE] Release Process

+1 (binding) -Kyle On Tue, Jan 17, 2017 at 11:17 PM, James Sirota wrote: > I made the revisions based on the discuss thread > > The document is available here: > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770 > > And is also attached for reference

Re: METRON-646 commit attribution

; Date: Mon Feb 27 21:05:46 2017 -0500 METRON-646 Add index templates to metron-docker (kylerichardson) closes apache/incubator-metron#441 commit 1e1b658b76f5c959bcebdf824372eea0d680d3f0 Author: Kyle Richardson <kylerichard...@apache.org> Date: Mon Feb 27 20:57:30 2017 -0500 Revert

Re: METRON-646 commit attribution

> > > > commit > > > > > has consequences. Not the least of which the scary force'd push. > > The > > > > > challenge here is that all forked repos during this period between > > the > > > > > wrong commit and the correction commit w

Re: Odd integration-test failures on Fedora/CentOS for RC5

h(outDir.getAbsolutePath()) >>> , new Path(queryDir.getAbsolutePath()) >>> , getTimestamp(0, pcapEntries) >>> , getTimestamp(pcapEntries.size()-1, pcapEntries) + 1 >>> , 10 >>>

Re: Odd integration-test failures on Fedora/CentOS for RC5

tto Fowler <ottobackwa...@gmail.com> wrote: > > What command are you using to build? > > > >> On February 25, 2017 at 17:40:20, Kyle Richardson >> (kylerichards...@gmail.com) wrote: >> >> Tried with Oracle JDK and got the same result. I w

Re: Files modified after building

That's my guess. I noticed the same thing. -Kyle > On Feb 25, 2017, at 8:31 PM, Otto Fowler wrote: > > Changes not staged for commit: > > (use "git add ..." to update what will be committed) > > (use "git checkout -- ..." to discard changes in working directory) >

Re: Odd integration-test failures on Fedora/CentOS for RC5

/pcap_pcap_1488155239282312000_0_pcap-9-1488153894,hdfs://node1:8020/apps/metron/pcap/pcap_pcap_1488155395170343000_0_pcap-9-1488153894 It feels like I'm missing something stupid. On Sun, Feb 26, 2017 at 8:53 PM, Kyle Richardson <kylerichards...@gmail.com> wrote: > Ok... Seems there is a d

METRON-646 commit attribution

Ok, so here's the story... Ryan was nice enough to commit my recent PR and for whatever reason my github username but not my email address appears in the commit author (see below). commit 41fc0ddc9881d9cfdd8bae129c0bb7800a116d4c Author: kylerichardson Date: Mon Feb 27 11:38:55 2017 -0600

Re: log parsers-

I have a working code for the ASA piece (METRON-363). Just finishing up some edge case testing. I'll submit a PR for it within your 2 week timeframe. Thanks, Kyle > On Sep 14, 2016, at 6:58 PM, Satish Abburi wrote: > > > Thanks, timelines are 2 weeks from now.

Re: [DISCUSS] Parsing messages without IP addresses

t IP address only really > means you won't be able to enrich based on THAT field, but could enrich on > other fields. > > I'd say leave them out if they aren't part of the format. It sounds like > some ASA events will have them and others won't, right? > On Sun, Sep 18,

[DISCUSS] Parsing messages without IP addresses

All, I've run into an edge case while working on METRON-363 . There are some log events which do not contain IP addresses and thus cannot be fully normalized into the standard Metron JSON fields. What are folks thoughts on how to handle this

Re: [DISCUSS] Recurring community meetings to demo Metron features

Great idea. +1 on the agenda. Maybe half demo of latest features / half discussion on upcoming changes and new ideas. -Kyle On Wed, Sep 21, 2016 at 4:03 PM, zeo...@gmail.com wrote: > I'm in from CMU. Zoom and WebEx work well. > > Only suggestion would be a basic agenda (I.e.

Re: [ANNOUNCE] Metron Apache Community Demo Recording Nov4,2016

Great use case. Really pulls a lot of the pieces together for me. Thanks for sharing. -Kyle On Fri, Nov 4, 2016 at 8:03 PM, James Sirota wrote: > The recording is available at: > https://youtu.be/vOMZcudmlYg > > The meeting was a demonstration of the upcoming build. No

Re: [DISCUSS] Next Release Name

g 0.2.2 (via Ambari management pack), and putting the configs > back > > into zookeeper, copying the ES templates and Grok files back, and > > restarting your NiFi flows. I agree that we should automate most of this > > eventually, and we will, but I don't think this is necessari

Re: [DISCUSS] Next Release Name

release I think the changes were significant >> enough due to the storm & kafka upgrade to justify moving it to a non-point >> release. Generally point releases are reserved for patches or maintenance >> releases. I think this release is more than just a maintenance release

Re: [DISCUSS] Next Release Name

I'm a little late to the party but thought I would go ahead and throw my two cents into the mix. I share the concern around an upgrade / migration path. While I would love to see the BETA dropped sooner than later, to me, this is a game changer for people implementing Metron. I think there is a

Re: HDFS Compression

Possibly naive question... Has there been past discussion on the use of avro for the data in HDFS? -Kyle On Tue, Oct 11, 2016 at 4:30 PM, Matt Foley wrote: > Some of the things that are desirable to do with stored data (including > those mentioned by others below): > - Use it

Re: [DISCUSS] Improving quick-dev

Sorry, I'm a bit late to the party on this one :). +1 on the four builds Nick described. Each would be useful and purpose built. I like the idea of using Docker, especially for local development and quick testing. Has anyone explored this? I'm envisioning very specific containers so you could

Re: [MENTORS] Release Maturity

I'm +1 on a meeting to discuss the backlog and would suggest, to be considered, each JIRA should have a clear description. I think that 72 hours is good for final adds to an upcoming release. I personally like the idea of having more visibility on which JIRAs are "up next" to help me figure out

Re: new committer: Kyle Richardson

l tool set [2] for commits. Not sure how many > are > > using it, but I personally use it all the time. > > > > [1] https://github.com/apache/incubator-metron/pull/330 > > [2] https://github.com/nickwallen/metron-commit-stuff > > > > > > > > On Wed,

Re: new committer: Kyle Richardson

tee (PPMC) for Apache Metron > (Incubating) > has asked Kyle Richardson to become a committer and we are pleased > to announce that they have accepted. > > > Being a committer enables easier contribution to the > project since there is no need to go via the patch > submission p

Re: Jira Rights

Any chance I could also get rights to assign myself issues in JIRA? If it's for PMC and committers only I totally understand. -Kyle On Mon, Oct 31, 2016 at 8:08 PM, James Sirota wrote: > Please sign up for the Jira and I will assign you the rights > > 28.10.2016, 19:35,

Re: [DISCUSS] Next Release Content

I'd like to selfishly add METRON-363 into consideration for the next release. This is complete pending final review and merge. -Kyle On Wed, Nov 2, 2016 at 5:00 PM, Casey Stella wrote: > Hey Jon, > > Regarding the

Re: Jira Rights

Thanks, James! My JIRA username is kylerichardson. -Kyle On Wed, Nov 2, 2016 at 10:28 AM, James Sirota <jsir...@apache.org> wrote: > Hi Kyle, you can have contributor privileges. Please sign up for the Jira > and I'll add you. Does anyone else want that too? > > 31.10.

Re: [GitHub] incubator-metron issue #276: METRON-363 Fix Cisco ASA Parser

Thanks, Otto! You're a genius. I'm at a loss for why this broke the integration tests. For me, it seems to have broke all of the integration tests which makes me think it broke some piece of the underlying framework. The big change seems to have been with the move to Storm 1.x but I can't say for

Re: Jira Rights

Make sure you sign up for the jira > > 02.11.2016, 08:00, "Kyle Richardson" <kylerichards...@gmail.com>: > > Thanks, James! My JIRA username is kylerichardson. > > > > -Kyle > > > > On Wed, Nov 2, 2016 at 10:28 AM, James Sirota <jsir...@apache.org&

Re: Issues with Quick Dev Installation

This error could be related to my PR that was merged today (METRON-492 ). I tested this successfully in a single node vm deployment, but not with vagrant. Perhaps there is something about vagrant that doesn't like the ansible local_action logic?

Re: Issues with Quick Dev Installation

for the fix. On Tue, Oct 11, 2016 at 8:42 PM, Kyle Richardson <kylerichards...@gmail.com> wrote: > This error could be related to my PR that was merged today (METRON-492 > <https://github.com/apache/incubator-metron/pull/302>). I tested this > successfully in a sing

Re: Issues with Quick Dev Installation

I like that idea. It seems much cleaner. Let me give it a try. On Tue, Oct 11, 2016 at 9:53 PM, Nick Allen <n...@nickallen.org> wrote: > Does it really need to sudo? Can we do something like "become: false" so it > doesn't try to sudo? > > On Oct 11, 2016 9:33 PM, &qu

Re: [ANNOUNCE] Metron Apache Community Demo Recording Oct14,2016

Great stuff! Very useful information. Thanks for hosting. -Kyle On Sat, Oct 15, 2016 at 3:57 AM, Yohann Lepage wrote: > Hi James, > > Thanks for the recording! > > Could you please also update the "Meeting Notes" page on the wiki with > the link to the recording? > >

Re: [PROPOSAL] up-to-date versioned documentation

Matt, thanks for pulling this together. I completely agree that we need to go all in on either cwiki or the README.md's. I think the wiki is poorly updated and can cause confusion for new users and devs. My preference is certainly for the README.md's. I like your approach but also agree that we

Re: [DISCUSS] Turning off indexing writers feature discussion

I'll second my preference for the first option. I think the ability to use Stellar filters to customize indexing would be a big win. I'm glad Matt brought up the point about data lake and CEP. I think this is a really important use case that we need to consider. Take a simple example... If I have

Re: [DISCUSS] Turning off indexing writers feature discussion

s >>>>>> for >>>>>>>>> those >>>>>>>>>> who need it. >>>>>>>>>> >>>>>>>>>> On Fri, Jan 13, 2017 at 10:39 AM, Nick Allen < >>>> n...@nickallen.org> >>>>

Re: Process for closing JIRAs

t; > > > > To the first question, mark the Jira as DONE. I don't believe we can do > > fix > > > version until the community agrees on the next version increment, which > > we > > > normally won't know until we're getting close to cutting a release and

Process for closing JIRAs

What's our current process for closing JIRAs once the github PR has been merged? Are we setting it to Done with a particular fix version? If a JIRA is a duplicate, are we marking it any particular way? Thanks, Kyle

Re: Tests failing due to new year

Created METRON-647 for tracking. -Kyle On Tue, Jan 3, 2017 at 3:49 PM, Kyle Richardson <kylerichards...@gmail.com> wrote: > ** This is causing all new PRs to fail Travis CI ** > > The rollover to the new year is causing unit test failures for some of our > parser clas

Re: Long-term storage for enriched data

gt; > definitely brought up the need to search HDFS from time to time. > > Things > > like Spark SQL, Hive, Oozie have been discussed, but Avro is new to > me > > I'll > > have to look into it. Are you able to summarize it's benefits? > > &g

Re: Long-term storage for enriched data

ts and document a required schema change in our > guides to creating custom enrichments. > > In ES we are currently doing one template per sensor which gives us that > overlapping field name (per sensor) flexibility. > > Jon > > On Fri, Jan 6, 2017, 12:33 PM Kyle Richards

Re: Long-term storage for enriched data

normalized the data is the > more fun I can have with it :) > > Jon > > On Fri, Jan 6, 2017, 4:26 PM Kyle Richardson <kylerichards...@gmail.com> > wrote: > >> You're right. I don't think it needs to account for everything. As I >> understand it, one of the

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

ezones across a > new > year boundary in addition to data replay. I would urge our main goal > here > to be idempotency. > > Best, > Mike > > On Tue, Jan 3, 2017 at 5:05 PM, Kyle Richardson < > kylerichards...@gmail.com> > wrote: &

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

gt; get > > a > > > more correct solution. > > > > > > I can take care of doing that today. Any objections to that solution? > > > > > > Justin > > > > > > On Wed, Jan 4, 2017 at 9:34 AM, Kyle Richardson < > > kylericha

Tests failing due to new year

** This is causing all new PRs to fail Travis CI ** The rollover to the new year is causing unit test failures for some of our parser classes. It looks like the issue in the same in all cases... We have hard coded a timestamp assertion but the original message does not contain the year and is now

Re: [DISCUSS] Release Process

I think this thread got commingled with the discussion on Coding Guidelines. The wiki page on the Release Process is at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=66854770. Overall, a really informative document. Thanks for pulling this together. Two questions: 1) I'm a

Re: [VOTE] Modify Bylaws

+1 (binding) -Kyle On Sat, Dec 17, 2016 at 10:40 AM, Nick Allen wrote: > Oops. My vote is binding. +1 > > On Fri, Dec 16, 2016 at 7:35 PM, Casey Stella wrote: > > > +1 binding > > On Fri, Dec 16, 2016 at 18:20 Matt Foley

Re: [DISCUSS] Metron IRC channel

I'll second the JIRA and Git integrations. I also like the meeting minutes feature. Maybe it's something we could think about for the future. -Kyle On Fri, Dec 16, 2016 at 3:57 PM, Casey Stella wrote: > I'll leave this open til monday and update the INFRA jira with the

[DISCUSS] Coding Guidelines

Couple of questions/comments: In 2.4, we talk about Javadoc and code comments but not too much about the user documentation. Should we, possibly in a section 4, give some recommendations on what should go into the README files versus on the wiki? This could also help the reviewer know if the

Re: Long-term storage for enriched data

This thread got me thinking... there are likely a fair number of use cases for searching and analyzing the output stored in HDFS. Dima's use case is certainly one. Has there been any discussion on the use of Avro to store the output in HDFS? This would likely require an expansion of the current

Re: [DISCUSS] Stepping down as release manager

+1 for Matt On Wed, Mar 22, 2017 at 10:23 AM, Justin Leet wrote: > Right now it's just support, not a vote. I assume, based on our past > practices, that there will be a separate [VOTE] thread. > > Justin > > On Wed, Mar 22, 2017 at 10:06 AM, Otto Fowler

Re: [VOTE] Final Board Resolution Draft V2

+1 (binding) On Mon, Mar 20, 2017 at 3:05 AM, James Sirota wrote: > > - Removed affiliations > - Added apache IDs where possible > - Removed committers and only left PPMC members > > Hope this version holds up. Please vote +1, -1, or 0 for neutral. The > vote will be open

Re: new committer: Jon Zeolla

Welcome Jon and Matt! Well deserved. Glad to have the opportunity to continue working with you both. -Kyle > On Mar 14, 2017, at 11:57 PM, Otto Fowler wrote: > > Congratulations Jon > > > On March 14, 2017 at 23:40:37, James Sirota (jsir...@apache.org) wrote: >

Re: [ANNOUNCE] Apache Metron (incubating) 0.3.1 is released

+1 to adding links to the website and downloads page from the top-level README in git. -Kyle On Fri, Mar 17, 2017 at 3:16 PM, Justin Leet wrote: > Why don't we just add a link to the downloads page in the main README.md? > Right now the "Obtaining Metron" section only

Re: [DISCUSS] SIDELOADING PARSERS: Packaging and Loading and Extensions [oh.my]

I like the direction of using NAR. The key benefits I see (there are others, but I picked by favorites): 1. Move away from the uber jars in Storm. Granted, this should be possible with modifications from the new Stellar classloader. My hope is that this would allow us to stop shading the parser

Re: [DISCUSS] SIDELOADING PARSERS: Archetype

Could the archetype be deployment agnostic? It's probably a little simplistic due to all the configs, but I like the NAR solution of simply drop in lib and restart services. On Fri, Mar 10, 2017 at 10:04 AM, Otto Fowler wrote: > As previously discussed here, I have been

Re: [VOTE] Final Board Resolution Draft

s passed through the system. >>> >>> RESOLVED, that the office of "Vice President, Apache Metron" be >>> and hereby is created, the person holding such office to >>> serve at the direction of the Board of Directors as the chair >>> of the Apa

[DISCUSS] New Stellar Functions

I have the need for a new Stellar function to perform string concatenation. I have it implemented but am curious about where new functions should live given the new capabilities around 3rd party Stellar function libraries. So, I guess my question is, should this function live in: 1) metron-common

Stellar field transformations cannot use input fields with dashes

So, I learned something the hard way today and thought I would share. I'm sure most of you already knew this but here goes. It turns out that, while you can use message field names with dashes in them (e.g. cs-host) you cannot perform Stellar operations with them. In my case, this was a field

Re: [DISCUSS] MPack components that don't support Kerberos

Just to clarify, we're saying when adding a service to the mpack that Kerberos support is required... but we're not saying that installing Metron requires a kerberized cluster correct? I think we should support it but should allow installation and use of Metron without it (for testing or other

Re: [VOTE] Cesey Stella for Metron VP

+1 (binding) On Mon, Mar 13, 2017 at 11:56 PM, Anand Subramanian < asubraman...@hortonworks.com> wrote: > +1 (non-binding) > > > > > On 3/14/17, 4:03 AM, "James Sirota" wrote: > > >This vote is to make Casey Stella our VP after graduation > > > >--- > >Thank

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

Solid work, Otto. I'm excited to see us move in this direction. It's an important step to making Metron a more user friendly platform. I agree with Matt. I think a PR for this piece is needed sooner rather than later so you don't continually have to rework it as we make more commits into master.

Re: [Discuss] SIDELOADING PARSERS: Parsers as components

system location ( hdfs in the end ). > I imagine us doing that through the UI. > > Nar’s *are* automatically unpacked at execution time. > > Nar’s are unpackaged into a ‘working’ area ( if required ), that working > area is the class path that is > loaded. > > > >