Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

> > > > > > On 1/13/17, 8:11 AM, "Billie Rinaldi" <bil...@apache.org> wrote: > > > > No, we can't host artifacts in a git repo, or on a website. It would > be > > like distributing a release that hasn't been voted upon. > >

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

it-submodules-for-maven- > artifacts-not-in-central/ > > Thanks, > > On Tue, Jan 31, 2017 at 11:41 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Hi Billie, > > > > Thanks for the feedback and info. I'm working on resolving this over

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

<bil...@apache.org> wrote: > On Tue, Jan 31, 2017 at 9:41 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Hi Billie, > > > > Thanks for the feedback and info. I'm working on resolving this over the > > next couple of days and cou

Re: [DISCUSS] Build Times are getting out of hand

I agree with this. I don't think we should switch to an alternate system until we find that we are absolutely incapable of eking out any further efficiency from the current setup. On Tue, Feb 7, 2017 at 8:04 AM, Casey Stella wrote: > I believe that some people use travis and

Re: [DISCUSS] Build Times are getting out of hand

t > Casey suggested." :) > > -D... > > > On Tue, Feb 7, 2017 at 10:45 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > I agree with this. I don't think we should switch to an alternate system > > until we find that we are absolutely i

Re: [DISCUSS] Next Release (0.3.1) Content

Hi Casey, My understanding from legal and our mentors is that as it currently stands, we would not have to wait. http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/201701.mbox/%3CCAF1jEfCGVGd9tVDJfRwoxFjEmwHD59DuXTdMu-_OJQwcLvryog%40mail.gmail.com%3E and

Re: [DISCUSS] How to do Sliding Windows in Profiler

Hey guys, 1. I'm going to Google that brandy analogy later. It sounds tasty :) 2. The option for reading/writing multiple profiles makes sense to me and provides much greater flexibility and efficiency. 3. My gut reaction to reading through this is "wow." This seems really complicated. I'm all

Re: custom date format required for snort, but not working

We decided at some point that given that there is an option in Snort to enable years in the timestamp that this was the best option for handling the dates. This should already be the default for Vagrant. On Tue, Feb 21, 2017 at 5:59 AM, Otto Fowler wrote: > ok - > > #

Re: [DISCUSS] Sketch Libraries

@Casey That's correct - the stream-lib library has an HLL+ implementation that not only works better for small data set sizes, but also scales to much larger cardinalities than the previous HLL algorithm. The DataSketches library appears to be a vanilla HLL implementation. On Wed, Feb 22, 2017 at

Re: [DISCUSS] Top domains enrichment config/extractor management

r, we should design > the management UI so that the enrichments can be added easily, with a > wizard to enable the use of the enrichment via stellar for a sensor > > On that topic, I think we are doing too much as part of our install. I > would argue that we shouldn't pre-l

Re: [DISCUSS] Top domains enrichment config/extractor management

members willing > to do that work. > > For example, think about the Homebrew project. They make it stupid simple > to add a new installable package. You don't have to know how Homebrew > works to contribute a package. The result is they have tons of packages > available. > > Does th

Re: [DISCUSS] Coding style via checkstyle

nfiguration. > > > > > > On February 21, 2017 at 17:06:09, Casey Stella (ceste...@gmail.com) > wrote: > > > > +1 to blanket reformat as well. > > > > On Tue, Feb 21, 2017 at 1:25 PM, Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > &g

Re: [DISCUSS] Coding style via checkstyle

+1 to a blanket reformat, failed build for improper formatting, and automated formatting. I strongly prefer to remove "thinking" from my code formatting and it has worked very well for me on large projects in the past. There is capability now in IntelliJ to work with Checkstyle as well.

[DISCUSS] Top domains enrichment config/extractor management

With the work committed in https://github.com/apache/incubator-metron/pull/445 and https://github.com/apache/incubator-metron/pull/432, we now have a robust and flexible means to import enrichment sources and transform their contents as they are inserted into HBase. One of the main motivators for

Re: [DISCUSS] 0.3.1 Release situation

+1 on RC5 from master/HEAD On Wed, Feb 22, 2017 at 10:16 AM, Nick Allen wrote: > ​+1 on re-cutting RC5 from head.​ We're going to have to go through the > same level of effort either way. Might as well get more value out of it. > > > On Wed, Feb 22, 2017 at 11:43 AM, Casey

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC4

Verified steps 0 and 1. Will work on the remaining steps as time permits. M On Fri, Feb 10, 2017 at 1:22 PM, Casey Stella wrote: > This is a call to vote on releasing Apache Metron 0.3.1-RC4 incubating > > Full list of changes in this release: >

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

ion > is don’t cheat. > > Also, there may be circumstances under which the Ambari-agent will detect > changes and re-write the latest version it knows of the config files, even > without a Save or Start action at the Ambari-server. I’m not sure of this > and need to check w

Re: [GitHub] incubator-metron issue #418: METRON-666 Fix javadoc doclint errors

How does Maven split tasks up these tasks? My only concern there is that we have some robust integration test infrastructure data setup that may not be thread-safe. On Tue, Jan 17, 2017 at 7:55 AM, jjmeyer0 wrote: > Github user jjmeyer0 commented on the issue: > >

Re: [PROPOSAL] up-to-date versioned documentation

Hey Matt, feel free to ping me. On Mon, Jan 16, 2017 at 1:39 PM, Matt Foley wrote: > I looked into the Falcon website and doxia over the weekend, and I’m > convinced that using the doxia-markdown plugin should make it dirt simple > to do what’s been discussed in this thread,

Re: [DISCUSS] Moving GeoIP management away from MySQL

I'm also in agreement on this. On Mon, Jan 16, 2017 at 2:11 PM, Nick Allen wrote: > +1 to using the Java API with the MMDB file provided by Maxmind. This is > what I had thought we were doing when we discussed this a few months back. > I'd rather use the Maxmind tools

Re: [PROPOSAL] up-to-date versioned documentation

t; it > > > doesn’t > > > correctly combine bullet lists subordinate to a numbered list. > > > > > > The upshot is that > > > • both code and bullet lists often lose their linebreaks, and > get > > > mus

Re: Metron Travis Build parameters

I can't think of any. I am +1 for adding it. On Tue, Feb 28, 2017 at 1:38 PM, Otto Fowler wrote: > 1395.36s$ time mvn -q -T 2C -DskipTests install && time mvn -q -T 2C > surefire:test@unit-tests && mvn -q surefire:test@integration-tests && time >

Re: [DISCUSS] Making adding new 3rd-party Stellar functions easier

Just so I'm clear Otto, when you discuss the UBER jar, this is a basic parser jar with the system libraries from Metron. And based on METRON-744, we would now be able to load parser jars that are created with the Maven archetypes developed in METRON-258 by loading them into a specific location in

Re: [DISCUSS] Upcoming Metron Build

we now have everything we wanted in there? > >> > >> 27.09.2016, 08:59, "David Lyle" <dlyle65...@gmail.com <javascript:;>>: > >> > I'd like to hold out for https://issues.apache.org/ > >> jira/browse/METRON-466. > >> > I'm tes

Re: [VOTE] Releasing Apache Metron 0.2.1BETA-RC1

Sweet! Thanks David. On Thu, Sep 29, 2016 at 4:18 PM, David Lyle wrote: > Sounds good to me. I can push up a pr straightaway. Testing the fix now. > > -D... > > > On Thu, Sep 29, 2016 at 4:17 PM, Casey Stella wrote: > > > Let's cancel the vote, someone

Re: [VOTE] Releasing Apache Metron 0.2.1BETA-RC1

David, is line 42 in incubator-metron/metron-deployment/inventory/quick-dev-platform/group_vars/all the only issue you see? On Thu, Sep 29, 2016 at 4:17 PM, Casey Stella wrote: > Let's cancel the vote, someone put up a PR to correct and cut a rc2. How > about that? > > On

[DISCUSS] Storm topology sideloading jars

As part of https://issues.apache.org/jira/browse/METRON-356 it is now possible to add hbase and hadoop conf to the Storm topology classpath. It is also desirable to expand this functionality to sideloading jars for Storm topologies. That way, users can add additional dependencies without having to

Re: [DISCUSS] Upcoming Metron Build

Also need this Jira to bump the version. https://issues.apache.org/jira/browse/METRON-398 On Tue, Sep 27, 2016 at 11:38 AM, James Sirota wrote: > We are preparing the next release (will be put up for a vote within the > next few days) with the following list of Jiras. Do

Re: [DISCUSS] Dockerize Metron

Any ideas on how debugging works when leveraging Docker? In spite of the classpath troubles, one of the benefits of the current single-JVM approach is that you can easily debug, set break points, etc within an IDE. Is that still doable? Seems like there might need to be some remote debugging magic

Re: [DISCUSS] Dockerize Metron

2016 at 1:15 PM, Casey Stella <ceste...@gmail.com> wrote: > > > > > I agree with that concern. Being able to debug a running topology has > > > helped in so many circumstances. Not sure how to accomplish this in a > > > dockerized environment. > > > > &

Re: [CANCELLED][VOTE] Releasing Apache Metron 0.2.1BETA-RC1

all it is. We just crossed is all. > >> > >> -D... > >> > >> On Thu, Sep 29, 2016 at 4:29 PM, Michael Miklavcic < > >> michael.miklav...@gmail.com> wrote: > >> > >> > Sweet! Thanks David. > >> > >

Re: new PPMC Member: David Lyle

Congrats David! On Fri, Sep 30, 2016 at 1:42 PM, James Sirota wrote: > The Podling Project Management Committee (PPMC) for Apache Metron > (Incubating) > has asked David Lyle to become a PPMC member and we are pleased > to announce that they have accepted. > > Being a

Re: new PPMC Member: Nick Allen

Congrats Nick! On Fri, Sep 30, 2016 at 1:43 PM, James Sirota wrote: > The Podling Project Management Committee (PPMC) for Apache Metron > (Incubating) > has asked Nick Allen to become a PPMC member and we are pleased > to announce that they have accepted. > > Being a

Re: Help with custom enrichment / parser

ot;fieldToTypeMap" : { > "ip_src_addr" : [ "malicious_ip" ], > "ip_dst_addr" : [ "malicious_ip" ] > }, > "config" : { }, > "triageConfig" : { > "riskLevelRules" : { }, >

Re: [DISCUSS] Metron next version rev

ecent PRs that > would > > > > qualify as patches? I'd just like to get a feel for how many of > those > > > > might exist. > > > > > > > > > > > > > > > > On Mon, Nov 14, 2016 at 6:58 PM, David Lyle <dlyle65...@gmail.

Re: [DISCUSS] Metron next version rev

that should all be worked out as we go. > > > On November 15, 2016 at 12:13:55, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'm a +1 on David and Nick's suggestions. 1 and 2 now, and let 3 happen > organically when the community has a need. > > On Tue, No

Re: [DISCUSS] Metron next version rev

Once 0.3.0 is > >> released, master is no longer the 0.3.0 branch. > >> > >> I recommend that we run 0.3.x patches off the 0.3.0 release branch and > >> rename master to 0.4.0. > >> > >> Thanks, > >> > >> -D... > >> > >> &

Re: Help with custom enrichment / parser

Tyler, Thanks for the interest in Metron and welcome to the community! :) Just curious, what type of environment are you running in? Full cluster or are you using the full-dev or quick-dev vagrant deployment vagrant scripts? Best, Mike Miklavcic On Thu, Nov 3, 2016 at 10:34 AM, Tyler Moore

Re: Help with custom enrichment / parser

a topic correct?? > > Thanks for all your help thus far! > > Regards, > > Tyler > > Regards, > > Tyler Moore > Software Engineer > Flyball Labs > > On Thu, Nov 3, 2016 at 2:42 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > >

Re: [DISCUSS] Next Release Name

Hi Jon, I have commented on 370 - https://issues.apache.org/jira/browse/METRON-370 Best, Mike On Wed, Nov 2, 2016 at 3:11 PM, zeo...@gmail.com wrote: > I personally would like to see the following things done before things > leave BETA: > (1) Address data integrity concerns

Re: [DISCUSS] Intentional processing delays

So, you want to queue up the data awaiting a key match on the enrichment data, up to a max timeout and/or buffer size? Seems like this should belong at the spout level to avoid buffer overflows, depending on how big the data sets are and how far apart the matching records/elements are spaced in

Re: [DISCUSS] Improving quick-dev

I think this may have come up in another PR already (have to look for it). But maybe we could maintain our flexibility in quick-dev by installing the sensors and not starting them until we need them. I think it's useful to have a quick "genuine" e2e testing environment that doesn't require running

Re: [VOTE] Coding Guidelines

Can someone clarify this point in merge requirements? "All merged patches must have 100% test coverage." On Fri, Dec 16, 2016 at 10:34 AM, James Sirota wrote: > I incorporated the changes to the coding guidelines from our discuss > thread. I'd like to get them voted on to

Re: [VOTE] Coding Guidelines

tion is a decent goal, but I don't like a code coverage req. > On Fri, Dec 16, 2016 at 13:48 Michael Miklavcic < > michael.miklav...@gmail.com> > > wrote: > > > Can someone clarify this point in merge requirements? > > > > "All merged patches must have

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

Any comment from the mentors on this? On Mon, Jan 9, 2017 at 2:00 PM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > This is a question primarily for the mentors. > > *Background* > metron-common is currently depending on the openSOC github repo for > hos

Re: [DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

-symbol. The other option is to repeat > "mentors" 3 times and either the mentors will appear or you will be eaten > by a grue. ;) > > On Wed, Jan 11, 2017 at 9:32 AM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Any comment from the mentors o

[DISCUSS][MENTORS] Hosting Kraken maven artifacts in incubator-metron git repo

Restarting this thread w/new subject This is a question primarily for the mentors. *Background* metron-common is currently depending on the openSOC github repo for hosting kraken artifacts. The original reason for this was that these jars are not hosted in Maven Central, and they were not

Re: Kafka error when upgrading to Metron 0.3.0

like it's just the unit tests that are failing for pcap backend. Was > able to build successfully using new came with -DskipTests option along > with the profile argument. > > Regards, > > Tyler Moore > IT Specialist > tyler.math...@yahoo.com > 248-909-2769 > > >

Re: [DISCUSS] Dev Guide and Committer Review Guide additions?

tecture needs to > include update methodology for semantic changes in parameter sets. > > I’m mulling such an architecture, but what do other people think? Is this > a valid need? > > Thanks, > --Matt > > On 1/12/17, 8:23 AM, "Michael Miklavcic" <michael.miklav...@gm

Re: [PROPOSAL] up-to-date versioned documentation

Casey, Matt - These guys are using doxia https://github.com/apache/falcon/tree/master/docs Honestly, I kind of like Spark's approach - https://github.com/apache/spark/tree/master/docs Mike On Thu, Jan 12, 2017 at 4:48 PM, Matt Foley wrote: > I’m ambivalent; I think we’d end

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

Hi Casey, Thanks for starting this thread. I believe you are correct in your assessment of the 4 options for updating configs in Metron. When using more than one of these options we can get into a split-brain scenario. A basic example is updating the global config on disk and using the

Re: [DISCUSS] Turning off indexing writers feature discussion

I like the flexibility and expressibility of the first option with Stellar filters. M On Thu, Jan 12, 2017 at 1:51 PM, Casey Stella wrote: > As of METRON-652 , we > will have decoupled the indexing configuration from the

Re: [DISCUSS] Dev Guide and Committer Review Guide additions?

Hi Otto, You make a great point. AFA RPM/MPack, we do have some work in the pipeline for streamlining things a bit with the RPM's and MPack code such that they will be used for performing the Metron install in the sandbox VM's rather than Ansible. (I'd search for the public Jiras and post them

Re: Process for closing JIRAs

To the first question, mark the Jira as DONE. I don't believe we can do fix version until the community agrees on the next version increment, which we normally won't know until we're getting close to cutting a release and can make an assessment of it being major or minor. On Mon, Dec 5, 2016 at

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

I also introduced a Clock object and testing mechanism back in METRON-235 - https://github.com/apache/incubator-metron/pull/156 Sample test utilizing the Clock object here -

Re: METRON-648 GrokWebSphereParserTest and BasicAsaParserTest are not 2017-safe

> > Granted, if the timezone is also unspecified, there is still a 24 hr > > period > > > of uncertainty, but it does seem that on Jan 3 2017 the preferred > > > interpretation of “Apr 15” would be Apr 15 2016, not 2017. > > > > > > Cheers,

Re: Kafka error when upgrading to Metron 0.3.0

2.18:test (integration-tests) on project metron-pcap-backend: There are test failures. Regards, Tyler Moore Software Engineer Phone: 248-909-2769 <(248)%20909-2769> Email: moore.ty...@goflyball.com On Thu, Jan 5, 2017 at 5:21 PM, Michael Miklavcic < michael.miklav...@gmail.com&g

[DISCUSS] Hosting Kraken maven artifacts in incubator-metron git repo

This is a question primarily for the mentors. *Background* metron-common is currently depending on the openSOC github repo for hosting kraken artifacts. The original reason for this was that these jars are not hosted in Maven Central, and they were not reliably available in the Kraken repo.

Re: [DISCUSS] Coding Guidelines

en (and > >>> vice versa) > >>> > > Objects are checked (instanceof) for appropriate type > before > >>> casting (use > >>> > > generics if possible) > >>> > > Pub

Re: [DISCUSS] Coding Guidelines

nsible, mpack, or (soon :-) docker, should be acceptable. > > > > Thanks, > > --Matt (who can’t run vagrant workably on the systems available to > me) > > > > > > On 12/21/16, 8:52 AM, "Michael Miklavcic" < > michael.mikl

Re: [DISCUSS] Metron IRC channel

Jira search On Fri, Dec 16, 2016 at 11:50 AM, Otto Fowler wrote: > Start with jira and git? > > > > On December 16, 2016 at 13:17:06, Casey Stella (ceste...@gmail.com) wrote: > > Hi all, > > Any ideas of what features we would like to add? The options are at: >

Re: [DISCUSS] Coding Guidelines

Were you thinking javadoc or something more? I wouldn't mind seeing us produce a javadoc site, if we aren't already doing so. On Dec 20, 2016 9:25 AM, "zeo...@gmail.com" wrote: > Regarding documentation - while I'm not a huge fan of that approach (I > would prefer to see

Re: [DISCUSS] Metron VP

+1 to Casey (nonbinding) On Sun, Mar 12, 2017 at 10:20 PM, James Sirota wrote: > I would like to propose that Casey Stella be our VP upon graduation. I > think has been the most outspoken proponent of the "Apache way" on our > project and has made very significant

Re: [DISCUSS] New Stellar Functions

Hey Kyle, It probably belongs here - https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/StringFunctions.java There is an existing JOIN function for strings - might this suit your needs? I didn't see a unit test

Re: Kerberos changes affected quick-dev and full-dev

Awesome, thanks David! On Tue, Apr 4, 2017 at 7:16 AM, Casey Stella wrote: > Thanks David! > > On Mon, Apr 3, 2017 at 8:43 PM, David Lyle wrote: > > > I've pushed a new Vagrant image for Quick Dev. You should be asked to > > update the box the next

Re: [DISCUSS][PROPOSAL] Acceptance Tests

"I don't think acceptance tests should loosely associate with real uses, but they should be free to delve into weird non-happy-pathways." Not following - are you saying they should *tightly* associate with real uses and additonally include non-happy-path? On Fri, Mar 3, 2017 at 12:57 PM, Casey

Re: [DISCUSS] Wiki use and migration of docs

Just to clarify the point about migrating from the wiki - you'd like to see use cases, demos, cookbooks, etc. moved into the git repo? I'm +1 for this as it allows us to version the examples with each commit and release. A feature that is currently lacking and more difficult to accomplish on the

Re: [DISCUSS][PROPOSAL] Acceptance Tests

did exercise the > components similar to real-world use-cases. They should also be able to be > able to tread outside of the happy-path for those use-cases. > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > "I don't

Re: [GitHub] incubator-metron issue #507: METRON-819: Document kafka console producer par...

Can you try listing and applying acls with the root user instead of metron? On Fri, Apr 7, 2017 at 10:29 AM, nickwallen wrote: > Github user nickwallen commented on the issue: > > https://github.com/apache/incubator-metron/pull/507 > > I went through your