> (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update Ansible to pin to that release.
The problem with this approach, as I see it, is that it commits us to having separate releases for the Bro plugin. Which I am not sure if or how long it will take to come to a consensus on that. We also would need to invent that release process rather quickly. > (4) Revert PR #837 because as you pointed out this approach does not work well with releases. That would give us enough time to come up with a sensible approach and do that. I am kind of leaning towards (4) right now. The approach taken in this PR, doesn't work. Let's just revert it and give ourselves some time to come up with an approach that does work and has consensus. On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen <n...@nickallen.org> wrote: > Just to layout some other alternatives... > > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update > Ansible to pin to that release. > > (2) [Jon] Update Ansible to pin to a specific commit. > > (3) Wouldn't the submodule approach solve this? I imagine that if we use > a submodule, then all of the Bro plugin code will be contained directly in > each Metron release. We would just need to change that small bit of > Ansible code so that it uses the code directly (like before) instead of > going to Git and checking it out. > > (4) Revert PR #837 because as you pointed out this approach does not work > well with releases. That would give us enough time to come up with a > sensible approach and do that. > > > > > > > On Thu, Nov 16, 2017 at 10:21 AM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > >> The problem is that we're currently pinning to master and if something in >> the plugin breaks backward compatibility, our prior releases will be >> perpetually broken, which is why I suggest it blocks the upcoming release. >> >> The alternative would be to update ansible to pin to a specific commit or >> to make a release in apache/metron-bro-plugin-kafka sooner rather than >> later and pin to its branch/tag. That feels like a waste of time though, >> as the 2.5.2 upgrade is somewhat trivial. >> >> Jon >> >> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <n...@nickallen.org> wrote: >> >> > While I think that the Bro work is super valuable, Jon, I am not sure >> that >> > we need to block the next release for it. In my mind, the "theme" of >> the >> > next release is Metaalerts running on ES 2.x. I'd prefer to just stick >> > with that. >> > >> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs merged >> > and start cutting release candidates ASAP. That could be possible by >> end >> > of week based on the "big one" (METRON-1289) getting merged in last >> night. >> > >> > Of course, if you happen to get all the Bro work done in time, then >> > definitely let's include it in the next release. But I am wary of >> blocking >> > the release for that work. No need for you to rush through it. >> > >> > Just one man's opinion. Would like to hear feedback from more of the >> > community. >> > >> > >> > >> > On Thu, Nov 16, 2017 at 8:01 AM, zeo...@gmail.com <zeo...@gmail.com> >> > wrote: >> > >> > > The way master's full-dev is set up right now is non optimal for the >> bro >> > > plugin configuration, and I would like to complete the roadmap I >> outlined >> > > in my discuss thread before a release. I have a PR open against >> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin, >> and I >> > > expect it will take me until end of next week at the latest to have >> the >> > > rest of the work done to move us to 2.5.2, and to pin to a specific >> > package >> > > version. At that point I'm comfortable with a release. >> > > >> > > Jon >> > > >> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote: >> > > >> > > > I’ve been listening. Looks like there are still a number of major >> > issues >> > > > to be committed first, right? >> > > > The discussion on this thread constitutes sufficient engagement, I >> > think, >> > > > especially given the Subject line :-) >> > > > Would the folks working on the 6 issues listed by Nick care to >> suggest >> > a >> > > > cut-off date by which they’ll probably have those fixes in? >> > > > I’ll be happy to run the release process, if the community so >> wishes, >> > as >> > > > soon as those issues are committed. >> > > > >> > > > I guess I should, pro forma, send the list of commits already in >> since >> > > the >> > > > last release. I’ll do that today. >> > > > Also, if anyone wishes to raise a hand and propose additional >> commits >> > are >> > > > needed, please do so on this thread. >> > > > Thanks, >> > > > --Matt >> > > > >> > > > >> > > > On 11/15/17, 2:09 PM, "Casey Stella" <ceste...@gmail.com> wrote: >> > > > >> > > > I'd say that if a release is this imminent that we had better >> > notify >> > > > the >> > > > release manager who will make a release announcement, Nick. >> Matt, >> > > are >> > > > you >> > > > tuning in to this? >> > > > >> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen < >> n...@nickallen.org> >> > > > wrote: >> > > > >> > > > > Hi Guys - >> > > > > >> > > > > I want to follow-up on this discussion. It sounds like most >> > people >> > > > are in >> > > > > agreement with the general approach. >> > > > > >> > > > > A lot of people have been working hard on Metaalerts and >> > > > Elasticsearch. I >> > > > > have checked-in with those doing the heavy lifting and have >> > > compiled >> > > > a more >> > > > > detailed plan based on where we are at now. To the best of my >> > > > knowledge >> > > > > here is the plan of attack for finishing out this effort. >> > > > > >> > > > > (1) First, METRON-1289 needs to go in. This one was a >> fairly >> > big >> > > > effort >> > > > > and I am hearing that we are pretty close. >> > > > > >> > > > > (2) METRON-1294 fixes an issue in how field types are >> > looked-up. >> > > > > >> > > > > (3) METRON-1290 is next. While this may have been fixed in >> > > > M-1289, there >> > > > > may be some test cases we want from this PR. >> > > > > >> > > > > (4) METRON-1301 addresses a problem with the sorting logic. >> > > > > >> > > > > (5) METRON-1291 fixes an issue with escalation of >> metaalerts. >> > > > > >> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This >> > > > introduces the >> > > > > UI bits that depend on all the previous backend work. >> > > > > >> > > > > (7) At this point, we should have our best effort at running >> > > > Metaalerts >> > > > > on Elasticsearch 2.x. I propose that we cut a release here. >> > > > > >> > > > > (8) After we cut the release, we can introduce the work for >> ES >> > > 5.x >> > > > in >> > > > > METRON-939. I know we will need lots of help testing and >> > reviewing >> > > > this >> > > > > one. >> > > > > >> > > > > Please correct me if I am wrong. I will try and send out >> updates >> > > as >> > > > we >> > > > > make progress. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com < >> > zeo...@gmail.com >> > > > >> > > > wrote: >> > > > > >> > > > > > I agree, I think it's very reasonable to move in line with >> > Nick's >> > > > > > proposal. I would also suggest that we outline what the >> target >> > > > versions >> > > > > > would be to add in the METRON-777 components, since it has >> been >> > > > > functional >> > > > > > for a very long time but not reviewed and has some really >> > > rockstar >> > > > > > improvements. >> > > > > > >> > > > > > Jon >> > > > > > >> > > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler < >> > > > ottobackwa...@gmail.com> >> > > > > > wrote: >> > > > > > >> > > > > > > I think the ES cutover should be the start of the 0.5.x >> > series, >> > > > and we >> > > > > > > continue on with 0.4.x for the >> > > > > > > metadata improvements etc. We could chose to focus >> 0.5.x’s >> > > first >> > > > > > releases >> > > > > > > on not only ES but >> > > > > > > getting a handle on kibana and the mpack situation as >> well. >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic ( >> > > > > > > michael.miklav...@gmail.com) wrote: >> > > > > > > >> > > > > > > I agree with your proposal, Nick. I think having a >> > stabilizing >> > > > release >> > > > > > > prior to upgrading ES/Kibana makes sense. >> > > > > > > >> > > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen < >> > n...@nickallen.org >> > > > >> > > > wrote: >> > > > > > > >> > > > > > > > I would like to start a discussion around upcoming >> > releases. >> > > > We have >> > > > > a >> > > > > > > > couple separate significant tracks of work that we need >> to >> > > > reconcile >> > > > > in >> > > > > > > our >> > > > > > > > release schedule. >> > > > > > > > >> > > > > > > > (1) We have had (and have in review) a good number of >> bug >> > > fixes >> > > > > > required >> > > > > > > to >> > > > > > > > support Metaalerts on the existing Elasticsearch 2.x >> > > > infrastructure. >> > > > > > > > >> > > > > > > > >> > > > > > > > (2) We also have ongoing work to upgrade our >> infrastructure >> > > to >> > > > > > > > Elasticsearch 5.x, which will not be backwards >> compatible. >> > > > > > > > >> > > > > > > > >> > > > > > > > I would like to see a release that has our best work on >> ES >> > > 2.x >> > > > before >> > > > > > we >> > > > > > > > migrate to 5.x. I would propose the following. >> > > > > > > > >> > > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x >> > > > > > > > >> > > > > > > > Release N+2: Cut-over to ES 5.x >> > > > > > > > >> > > > > > > > >> > > > > > > > (Q) Is it worth cutting a separate release for ES 2.x? >> Is >> > > > there a >> > > > > > better >> > > > > > > > way to handle the cut-over to 5.x? >> > > > > > > > >> > > > > > > >> > > > > > -- >> > > > > > >> > > > > > Jon >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > >> > > Jon >> > > >> > >> -- >> >> Jon >> > >
