For #3 we would need people to recursively clone our repo to pull in submodules properly. I work with other projects that do that and it is perpetually an issue for new users. I know it doesn't seem like a big deal, but it is definitely a deterrent to some who forget this step or use old instructions that don't include it.
I'm interested to see how other apache projects handle releases for multiple repos. I honestly think we can have a much lower bar for a release on a secondary repo, but I don't know if that's acceptable. Regarding 4, we could also fix forward. I already have a solution that could be very slightly repurposed in my full-dev testing instructions here <https://github.com/apache/metron-bro-plugin-kafka/pull/3>. Yes, I still prefer #5 but I'll stop talking so much and let also chime in. =) Jon On Thu, Nov 16, 2017 at 10:40 AM Nick Allen <n...@nickallen.org> wrote: > > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update > Ansible to pin to that release. > > The problem with this approach, as I see it, is that it commits us to > having separate releases for the Bro plugin. Which I am not sure if or how > long it will take to come to a consensus on that. We also would need to > invent that release process rather quickly. > > > (4) Revert PR #837 because as you pointed out this approach does not work > well with releases. That would give us enough time to come up with a > sensible approach and do that. > > I am kind of leaning towards (4) right now. The approach taken in this PR, > doesn't work. Let's just revert it and give ourselves some time to come up > with an approach that does work and has consensus. > > > > > > > > On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen <n...@nickallen.org> wrote: > > > Just to layout some other alternatives... > > > > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update > > Ansible to pin to that release. > > > > (2) [Jon] Update Ansible to pin to a specific commit. > > > > (3) Wouldn't the submodule approach solve this? I imagine that if we use > > a submodule, then all of the Bro plugin code will be contained directly > in > > each Metron release. We would just need to change that small bit of > > Ansible code so that it uses the code directly (like before) instead of > > going to Git and checking it out. > > > > (4) Revert PR #837 because as you pointed out this approach does not work > > well with releases. That would give us enough time to come up with a > > sensible approach and do that. > > > > > > > > > > > > > > On Thu, Nov 16, 2017 at 10:21 AM, zeo...@gmail.com <zeo...@gmail.com> > > wrote: > > > >> The problem is that we're currently pinning to master and if something > in > >> the plugin breaks backward compatibility, our prior releases will be > >> perpetually broken, which is why I suggest it blocks the upcoming > release. > >> > >> The alternative would be to update ansible to pin to a specific commit > or > >> to make a release in apache/metron-bro-plugin-kafka sooner rather than > >> later and pin to its branch/tag. That feels like a waste of time > though, > >> as the 2.5.2 upgrade is somewhat trivial. > >> > >> Jon > >> > >> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <n...@nickallen.org> wrote: > >> > >> > While I think that the Bro work is super valuable, Jon, I am not sure > >> that > >> > we need to block the next release for it. In my mind, the "theme" of > >> the > >> > next release is Metaalerts running on ES 2.x. I'd prefer to just > stick > >> > with that. > >> > > >> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs > merged > >> > and start cutting release candidates ASAP. That could be possible by > >> end > >> > of week based on the "big one" (METRON-1289) getting merged in last > >> night. > >> > > >> > Of course, if you happen to get all the Bro work done in time, then > >> > definitely let's include it in the next release. But I am wary of > >> blocking > >> > the release for that work. No need for you to rush through it. > >> > > >> > Just one man's opinion. Would like to hear feedback from more of the > >> > community. > >> > > >> > > >> > > >> > On Thu, Nov 16, 2017 at 8:01 AM, zeo...@gmail.com <zeo...@gmail.com> > >> > wrote: > >> > > >> > > The way master's full-dev is set up right now is non optimal for the > >> bro > >> > > plugin configuration, and I would like to complete the roadmap I > >> outlined > >> > > in my discuss thread before a release. I have a PR open against > >> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin, > >> and I > >> > > expect it will take me until end of next week at the latest to have > >> the > >> > > rest of the work done to move us to 2.5.2, and to pin to a specific > >> > package > >> > > version. At that point I'm comfortable with a release. > >> > > > >> > > Jon > >> > > > >> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <ma...@apache.org> wrote: > >> > > > >> > > > I’ve been listening. Looks like there are still a number of major > >> > issues > >> > > > to be committed first, right? > >> > > > The discussion on this thread constitutes sufficient engagement, I > >> > think, > >> > > > especially given the Subject line :-) > >> > > > Would the folks working on the 6 issues listed by Nick care to > >> suggest > >> > a > >> > > > cut-off date by which they’ll probably have those fixes in? > >> > > > I’ll be happy to run the release process, if the community so > >> wishes, > >> > as > >> > > > soon as those issues are committed. > >> > > > > >> > > > I guess I should, pro forma, send the list of commits already in > >> since > >> > > the > >> > > > last release. I’ll do that today. > >> > > > Also, if anyone wishes to raise a hand and propose additional > >> commits > >> > are > >> > > > needed, please do so on this thread. > >> > > > Thanks, > >> > > > --Matt > >> > > > > >> > > > > >> > > > On 11/15/17, 2:09 PM, "Casey Stella" <ceste...@gmail.com> wrote: > >> > > > > >> > > > I'd say that if a release is this imminent that we had better > >> > notify > >> > > > the > >> > > > release manager who will make a release announcement, Nick. > >> Matt, > >> > > are > >> > > > you > >> > > > tuning in to this? > >> > > > > >> > > > On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen < > >> n...@nickallen.org> > >> > > > wrote: > >> > > > > >> > > > > Hi Guys - > >> > > > > > >> > > > > I want to follow-up on this discussion. It sounds like most > >> > people > >> > > > are in > >> > > > > agreement with the general approach. > >> > > > > > >> > > > > A lot of people have been working hard on Metaalerts and > >> > > > Elasticsearch. I > >> > > > > have checked-in with those doing the heavy lifting and have > >> > > compiled > >> > > > a more > >> > > > > detailed plan based on where we are at now. To the best of > my > >> > > > knowledge > >> > > > > here is the plan of attack for finishing out this effort. > >> > > > > > >> > > > > (1) First, METRON-1289 needs to go in. This one was a > >> fairly > >> > big > >> > > > effort > >> > > > > and I am hearing that we are pretty close. > >> > > > > > >> > > > > (2) METRON-1294 fixes an issue in how field types are > >> > looked-up. > >> > > > > > >> > > > > (3) METRON-1290 is next. While this may have been fixed > in > >> > > > M-1289, there > >> > > > > may be some test cases we want from this PR. > >> > > > > > >> > > > > (4) METRON-1301 addresses a problem with the sorting > logic. > >> > > > > > >> > > > > (5) METRON-1291 fixes an issue with escalation of > >> metaalerts. > >> > > > > > >> > > > > (6) That leads us to Raghu's UI work in METRON-1252. This > >> > > > introduces the > >> > > > > UI bits that depend on all the previous backend work. > >> > > > > > >> > > > > (7) At this point, we should have our best effort at > running > >> > > > Metaalerts > >> > > > > on Elasticsearch 2.x. I propose that we cut a release here. > >> > > > > > >> > > > > (8) After we cut the release, we can introduce the work > for > >> ES > >> > > 5.x > >> > > > in > >> > > > > METRON-939. I know we will need lots of help testing and > >> > reviewing > >> > > > this > >> > > > > one. > >> > > > > > >> > > > > Please correct me if I am wrong. I will try and send out > >> updates > >> > > as > >> > > > we > >> > > > > make progress. > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Mon, Nov 6, 2017 at 1:03 PM, zeo...@gmail.com < > >> > zeo...@gmail.com > >> > > > > >> > > > wrote: > >> > > > > > >> > > > > > I agree, I think it's very reasonable to move in line with > >> > Nick's > >> > > > > > proposal. I would also suggest that we outline what the > >> target > >> > > > versions > >> > > > > > would be to add in the METRON-777 components, since it has > >> been > >> > > > > functional > >> > > > > > for a very long time but not reviewed and has some really > >> > > rockstar > >> > > > > > improvements. > >> > > > > > > >> > > > > > Jon > >> > > > > > > >> > > > > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler < > >> > > > ottobackwa...@gmail.com> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > I think the ES cutover should be the start of the 0.5.x > >> > series, > >> > > > and we > >> > > > > > > continue on with 0.4.x for the > >> > > > > > > metadata improvements etc. We could chose to focus > >> 0.5.x’s > >> > > first > >> > > > > > releases > >> > > > > > > on not only ES but > >> > > > > > > getting a handle on kibana and the mpack situation as > >> well. > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > On November 6, 2017 at 12:48:45, Michael Miklavcic ( > >> > > > > > > michael.miklav...@gmail.com) wrote: > >> > > > > > > > >> > > > > > > I agree with your proposal, Nick. I think having a > >> > stabilizing > >> > > > release > >> > > > > > > prior to upgrading ES/Kibana makes sense. > >> > > > > > > > >> > > > > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen < > >> > n...@nickallen.org > >> > > > > >> > > > wrote: > >> > > > > > > > >> > > > > > > > I would like to start a discussion around upcoming > >> > releases. > >> > > > We have > >> > > > > a > >> > > > > > > > couple separate significant tracks of work that we > need > >> to > >> > > > reconcile > >> > > > > in > >> > > > > > > our > >> > > > > > > > release schedule. > >> > > > > > > > > >> > > > > > > > (1) We have had (and have in review) a good number of > >> bug > >> > > fixes > >> > > > > > required > >> > > > > > > to > >> > > > > > > > support Metaalerts on the existing Elasticsearch 2.x > >> > > > infrastructure. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > (2) We also have ongoing work to upgrade our > >> infrastructure > >> > > to > >> > > > > > > > Elasticsearch 5.x, which will not be backwards > >> compatible. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > I would like to see a release that has our best work > on > >> ES > >> > > 2.x > >> > > > before > >> > > > > > we > >> > > > > > > > migrate to 5.x. I would propose the following. > >> > > > > > > > > >> > > > > > > > Release N+1: Introduce Metaalerts running on ES 2.x > >> > > > > > > > > >> > > > > > > > Release N+2: Cut-over to ES 5.x > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > (Q) Is it worth cutting a separate release for ES 2.x? > >> Is > >> > > > there a > >> > > > > > better > >> > > > > > > > way to handle the cut-over to 5.x? > >> > > > > > > > > >> > > > > > > > >> > > > > > -- > >> > > > > > > >> > > > > > Jon > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > -- > >> > > > >> > > Jon > >> > > > >> > > >> -- > >> > >> Jon > >> > > > > > -- Jon
