[DISCUSS] Release?

Is it about time for a release? I know we got some substantial performance changes in since the last release. I think we might have a justification for a release. Casey

Re: [DISCUSS] Release?

Is this what you mean Otto? https://github.com/apache/metron/blob/24822dddc68c264f59723f5e17d423cd497f6807/dev-utilities/release-utils/validate-jira-for-release On Wed, May 9, 2018 at 9:52 AM, Casey Stella wrote: > I wasn't aware we had a script for that..is that in >

Re: [DISCUSS] Release?

I get the following output (incidentally, I'm not sure if this is ok or not, but I noticed that this script pulled every tag and branch for any and all remotes I had defined in my local git repo) ~/devprojects/metron/dev-utilities/release-utils$ ./validate-jira-for-release --version=0.5.0

Re: [DISCUSS] Release?

Something like this might be more digestible for these purposes. $git log --pretty="%cr %s" tags/apache-metron-0.4.2-release..HEAD 88 minutes ago METRON-1530 Default proxy config settings in metron-contrib need to be updated (sardell via merrimanr) closes apache/metron#998 5 days ago METRON-1545

Re: [DISCUSS] Release?

+1 On Wed, May 9, 2018 at 9:13 AM, Casey Stella wrote: > Is it about time for a release? I know we got some substantial performance > changes in since the last release. I think we might have a justification > for a release. > > Casey >

Re: [DISCUSS] Release?

Definitely +1, with the Solr pieces going in too, does it make sense to bump the version to 0.5? On 9 May 2018 at 16:18, Michael Miklavcic wrote: > +1 > > On Wed, May 9, 2018 at 9:13 AM, Casey Stella wrote: > > > Is it about time for a release?

Re: [DISCUSS] Release?

Can you run the issues included script and post that for us to see? On May 9, 2018 at 11:14:11, Casey Stella (ceste...@gmail.com) wrote: Is it about time for a release? I know we got some substantial performance changes in since the last release. I think we might have a justification for a

Re: [DISCUSS] Release?

I wasn't aware we had a script for that..is that in dev-utilities/release-utils? On Wed, May 9, 2018 at 11:41 AM Otto Fowler wrote: > Can you run the issues included script and post that for us to see? > > > On May 9, 2018 at 11:14:11, Casey Stella (ceste...@gmail.com)

Re: Streaming Machine Learning use case

Hi Simon, That's correct. Apache SAMOA. Not any specific algorithm at this stage. Just the idea of being able to use streaming supervised learning without being worried of training cycle is interesting to me. The fact that it is closed to Metron from technology perspective made me wonder to see

Re: [DISCUSS] Release?

Ok, THAT is a beautifully useful command, Nick. We should totally have that as part of the scripts under dev-utilities. On Wed, May 9, 2018 at 12:14 PM Nick Allen wrote: > Something like this might be more digestible for these purposes. > > $git log --pretty="%cr %s"

Re: [DISCUSS] Release?

Oh, yes definitely (METRON-939) That one was a huge effort. On Wed, May 9, 2018 at 1:18 PM, zeo...@gmail.com wrote: > We should also mention the Upgrade of ElasticSearch and Kibana > > Jon > > On Wed, May 9, 2018 at 12:49 PM Nick Allen wrote: > > > Oh,

Re: [DISCUSS] Release?

I agree that it's probably time (more likely, overdue) for a release. Based off of looking at all of those changes I would also suggest going to at least 0.5.x. It probably makes sense to take a look at Upgrading.md (and related docs) to

Re: [DISCUSS] Release?

The next part of the conversation would be, what should the version number be? To help with that, I have tried to summarize the changes in the release. Of course, this is going to be heavily biased towards my own interests, so please feel free to chime in if I have missed anything. -

Re: [DISCUSS] Pcap panel architecture

This looks really great and gets me excited to maybe revisit some old conversations about PCAP capture in Metron. The only thing that I think it's missing is the ability to filter using bpf. I think the same thing can technically be accomplished by using packet_filter and I wouldn't throw a fit

Re: [DISCUSS] Release?

Oh, and also the Solr work that is currently in a feature branch. We would have to get the work finished up and merged though. Sounds like we are real close on that. On Wed, May 9, 2018 at 12:47 PM, Nick Allen wrote: > The next part of the conversation would be, what

Re: [DISCUSS] Release?

Good call - I thought that made our last release, but this would be the 2nd follow-on from when Nick originally posed a breakdown 4 months ago METRON-939: Upgrade ElasticSearch and Kibana (mmiklavc via mmiklavc) closes apache/metron#840

Re: [DISCUSS] Release?

I went through the old thread [1] that Mike mentioned. I wanted to make sure that I did not drop the ball on anything we discussed there. We proposed the following. - Release N+1: Introduce Metaalerts running on ES 2.x - Release N+2: Cut-over to ES 5.x What actually happened? -

Re: [DISCUSS] Release?

One item we haven't gotten around to was redoing the index names to use a metron_ prefix. I'm the one that pushed the original DISCUSS thread on this, but haven't had a chance to advance it. Does anyone have any strong opinions on it? I originally thought it made sense to include alongside the

Re: [DISCUSS] Pcap panel architecture

Thanks for the feedback Jon. I'm am not as familiar with BPF filtering as you probably are. Do you have an idea of much effort would be involved in implementing this? I suspect this would be another PcapFilter (

Re: [DISCUSS] Pcap panel architecture

This looks like a pretty good start Ryan. Does the metadata endpoint cover this https://github.com/apache/metron/tree/master/ metron-platform/metron-api#the-pcapgettergetpcapsbyidentifiers-endpoint from the original metron-api? If so, then we would be able to deprecate the existing metron-api

Re: [DISCUSS] Release?

Thanks for the background, Jon. I'd like to see the recent plugin enhancements included in the next release. On Wed, May 9, 2018 at 2:34 PM, zeo...@gmail.com wrote: > Nick - that was this >

Re: [DISCUSS] Release?

I don't think either of those block the release IMO. In fact, I can think of good reasons to wait on the Solr work since the feature branch is still active and incomplete. Regarding the indexing prefix, I'm all for that but I wouldn't wait on it. We have a ton of functionality and the

Re: [DISCUSS] Release?

Jon - What about all the work to use Bro's packaging mechanism for the Bro Plugin for Kafka? Wasn't that done after the repo split? I actually can't find the commit for that in the plugin repo after a quick look. Should that be considered part of the next release? Or did that happen longer

Re: [DISCUSS] Release?

I think we should have that. On May 9, 2018 at 14:16:23, Michael Miklavcic (michael.miklav...@gmail.com) wrote: One item we haven't gotten around to was redoing the index names to use a metron_ prefix. I'm the one that pushed the original DISCUSS thread on this, but haven't had a chance to

Re: [DISCUSS] Release?

Nick - that was this and this , which just barely snuck it into the same release cycle as 0.4.2. The plugin has

Re: [DISCUSS] Pcap UI user requirements

We are limited by Yarn and MapReduce applications in the case of pause/resume - I could be wrong, but I don't think that's something that's supported unless you're talking about multiple MR jobs strung together.

Re: [DISCUSS] Release?

IMO, It would be nice to have, but I don't consider it a blocker for the release. Of course, if its something that we can knock out soon (this week?), then there would be no reason not to include it. Did you create a JIRA for this one so we can track it? On Wed, May 9, 2018 at 2:16 PM, Michael

Re: [DISCUSS] Pcap UI user requirements

I had a feeling it may be that way. Unless anyone else knows of a better approach, it's probably most reasonable to push that into a follow-on JIRA and not over-complicate the current activities. Jon On Wed, May 9, 2018 at 2:33 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > We

Re: [DISCUSS] Release?

I don't have a strong opinion. The ES upgrade alone is a massive feature. It could make it easier to include the index change I mentioned along with Solr as a follow-up. I think if we did split, we could arguably start on the next release with Solr almost immediately. On Wed, May 9, 2018, 12:40

Re: [DISCUSS] Pcap UI user requirements

A couple of thoughts on cluster overuse: * Definitely can't pause/resume MR jobs, unfortunately * The traditional approach to managing overuse of cluster resources and prioritization in Yarn is via the scheduler. I'd suggest rather than building this ourselves, we allow users to be associated

Re: [DISCUSS] Release?

+1 to 0.5.0 On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com wrote: > I agree that it's probably time (more likely, overdue) for a release. > Based off of looking at all of those changes I would also suggest going to > at least 0.5.x. > > It probably makes sense to take a look

Re: [DISCUSS] Release?

+1 to 0.5.0 On Wed, May 9, 2018 at 2:05 PM Nick Allen wrote: > +1 to 0.5.0 > > On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com wrote: > > > I agree that it's probably time (more likely, overdue) for a release. > > Based off of looking at all of those

Re: [DISCUSS] Release?

I'm also a +1 on 0.5.0. This is a fairly big release. On Wed, May 9, 2018 at 12:05 PM, Nick Allen wrote: > +1 to 0.5.0 > > On Wed, May 9, 2018 at 1:36 PM, zeo...@gmail.com wrote: > > > I agree that it's probably time (more likely, overdue) for a release. >

Re: [DISCUSS] Release?

Simon brought up the idea of including the Solr enhancements (currently in a feature branch) for the release. What are people's opinions on this? Is this something that is a blocker for the release? IMO, there is so much already in master waiting to be released that I don't see a need to

Re: [DISCUSS] Pcap UI user requirements

Regarding the prioritization, that is what I was thinking as well, I just wasn't as prescriptive with my suggestion. I did look for a java implementation and failed to find one (the closest I found was the Apache-licened bcc project ). Perhaps someone else's

Re: [DISCUSS] Pcap panel architecture

Now that we are confident we can run submit a MR job from our current REST application, is this the desired approach? Just want to confirm. Next I think we should map out what the REST interface will look like. Here are the endpoints I'm thinking about: GET /api/v1/pcap/metadata?basePath This

Re: [DISCUSS] Release?

We should also mention the Upgrade of ElasticSearch and Kibana Jon On Wed, May 9, 2018 at 12:49 PM Nick Allen wrote: > Oh, and also the Solr work that is currently in a feature branch. We would > have to get the work finished up and merged though. Sounds like we are >