Do we have any further discussion on this? Pardon me if I misstate anyone's position, but it seems like we have a couple people (Otto and Jon and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably a section of people like myself without a particular horse in the race.
It seems like we need to come to some sort of consensus so that we can get the release bus moving again, and right now it seems like (a) is gathering more explicit support. Do we have a compelling reason to not do (a)? To be honest, my main worry is more "If we do (a) are we going to be miserable if we need to iterate or adjust?" I'm not seeing anything that suggests anything too terrible, so unless we see some more discussion, I suggest we move forward with (a). On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com <zeo...@gmail.com> wrote: > I would prefer a, but I was initially thinking of doing the plugin first > and then get in the two PRs out to use this new tag, which are already +1'd > and just waiting on this conversation. For reference, > https://github.com/apache/metron/pull/847 and > https://github.com/apache/metron-bro-plugin-kafka/pull/4 > > Jon > > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ottobackwa...@gmail.com> wrote: > > > It seems to me, as I believe I have stated before that a) feels like the > > proper way to handle this. It is how I have seen other projects like > NiFi > > handle things as well. > > > > > > > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote: > > > > Okay, looking at this from the perspective of making a release: > > > > > > > > We have two choices: > > > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of > > metron-bro-plugin-kafka, at the same time and using the same process > > (modulo the necessary) as Metron. This is dirt simple. > > > > b) I or someone needs to: > > > > - open a jira, > > > > - add the submodule to the Metron code tree, > > > > - possibly (optionally) add build mechanism to the maven poms, and > > > > - document as much as we think appropriate regarding what it is, how > to > > build it, and how to update it, > > > > and commit that before the 0.4.2 release. > > > > > > > > What is the will of the community? > > > > Thanks, > > > > --Matt > > > > > > > > From: Nick Allen <n...@nickallen.org> > > Reply-To: "dev@metron.apache.org" <dev@metron.apache.org> > > Date: Tuesday, November 28, 2017 at 9:09 AM > > To: "dev@metron.apache.org" <dev@metron.apache.org> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for > Bro' > > > > > > > > I'll add a bit to Jon's technical comments. > > > > > > > > * We only created a separate repo because it was a technical requirement > to > > leverage the bro-pkg mechanism. > > > > * Leveraging the new bro-pkg mechanism has many advantages as outlined by > > Jon. > > > > * Enabling the bro-pkg mechanism is backwards compatible. I can install > the > > plugin exactly how we use to. > > > > > > > > While I agree with Jon's technical comments, I disagree with the > > non-technical ones. > > > > > > > > (1) I do not want to change our release management process. While we > needed > > to make a new repo (a technical change), I did not want that to change > how > > we operate as a community (our procedures, policies, versioning and > release > > cycles). > > > > > > > > (2) I see no value in adopting a separate release management process for > > the Bro plugin alone. Having a separate release process does not make the > > plugin *more* available to the Bro community. > > > > > > > > (3) I also see no value in positioning the plugin to be spun-out of the > > Metron project. It is a part of Metron and I want to see it benefit and > > evolve "the Apache-way". > > > > > > > > In my mind, the best way to accommodate the additional repo, while > > minimizing changes to our release management process, is to treat the new > > repo as a submodule. I fail to see significant downsides to this > approach. > > A few extract command-line options do not seem overly onerous to me. > > > > > > > > Many thanks go to Jon for all the hard work he has put in enhancing the > > plugin. > > > > > > > > > > > > On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com <zeo...@gmail.com> > > wrote: > > > > In an attempt to keep this from becoming unbearably long, I will try to > > keep my responses short, but I would be happy to elaborate. That's a > fairly > > good timeline and summary, but here are some clarifications in > > corresponding order: > > > > > > > > - The plugin history is quite short and you can probably get a good bit > of > > context just by looking at the commits. > > > > - The plugin is only useful to the bro community, but it is rather > popular. > > > > - The Bro team created the idea of bro packages, which can include bro > > plugins, bro scripts, or BroControl plugins. So, instead of having a > > 'plugins' repo, they moved to have a 'packages' repo which is by default > > referenced by a bro-pkg tool they wrote for package management. > > > > - I believe I kicked this off (or at least I did in my head) when I > started > > complaining about the plugin divergence that occurred due to the move to > > bro/plugins (the right move at the time), but Metron's use of a local > > directory that hadn't been kept up to date. My current efforts are an > > attempt to make sure this doesn't happen again, and to take advantage of > > the bro-pkg benefits. > > > > - The gap between ~3/31 and actual progress on 11/12 is completely on me > - > > I had intentions of doing this work sooner but failed to do so. > > > > - You can most definitely still install/use the bro plugin without using > > bro-pkg. In fact, the README in my PR still has the instructions on how > to > > do so. > > > > > > > > Q1: The simple explanation is that the only thing that makes a plugin a > bro > > package is the inclusion of a bro-pkg.meta file, and it includes a > > build_command which could easily be manually performed to install by hand > > (assuming dependencies are met). > > > > > > > > I've worked with other projects that use submodules and while I'm fine > > discussing it, I suggest that we don't implement it. I put together a > quick > > example of why here[1], using the bro project as an example since it's > top > > of mind. > > > > > > > > Q2: I think the answer to Q1 answers this. There is absolutely nothing > > stopping a git clone && cd $dir && configure && make && make install, but > > using bro-pkg to install/load takes into account dependencies and unit > > tests when it is loaded (and thus fails early and more intuitively). It > > only must be a separate repo (or, more technically correct, a git branch > > that includes only the package) because of how bro-pkg works. If you'd > like > > to get an idea of how this would work in application for Bro users, you > can > > see my test instructions here (specifically step #3). If a 0.1 tag gets > > pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is > > just to remove user interaction, for an ansible spin-up). > > > > > > > > > > > > 1: To clone the Bro git repo, you must run `git clone --recursive > > https://github.com/bro/bro` <https://github.com/bro/bro> (note the > > --recursive). Not too big of a deal, > > but requires that you remember it and existing instructions/blog posts > may > > give users inaccurate steps. Let's make this worse and try to checkout > > their latest release, v2.5.2, and automatically update the submodules > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails > > because aux/plugins (https://github.com/bro/plugins) was removed since > > their latest release. Okay, we can work around this using `git checkout > > v2.5.2` and then remember to `git submodule update` every time you > checkout > > a release or branch. But because they have nested submodules, we actually > > need to run `git submodule update --recursive`. I can't imagine opting > into > > a workflow anything like this. There are other options as well, such as > git > > subtrees, but those I am less familiar with. > > > > > > > > Jon > > > > > > > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > I am not sure that our use of the plugin necessarily equates to it being > > implicitly coupled to Metron. It seems like the Right Thing To Do, esp. > > for an Apache project would be to make this available for use by the > > greater bro community. > > Unless we expect to do extensive iterative work on the plugin, which > would > > then make the decision to spin it out now premature. > > > > Then again, I might be wrong ;) > > > > > > On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote: > > > > [Please pardon me that the below is a little labored. I’m trying to > > understand the implications for both release and use, which requires some > > explanation as well as the two questions needed. Q1 and Q2 below are > > probably the same question, asked in slightly different contexts. Please > > consider them together.] > > > > So this made me go back and look at the history that caused us to put the > > bro plugin in a separate repo. As best I can see, this was in > > https://issues.apache.org/jira/browse/METRON-813 , which cites an email > > discussion thread. Also please see > > https://issues.apache.org/jira/browse/METRON-883 for background on the > > plugin itself. > > > > As best I can assemble the many bits brought up in the threads, the > reasons > > to put it in a separate repo was: > > - The plugin was thought to be useful to multiple clients of bro and > kafka, > > including Storm and Spark, as well as Metron. > > - Originally the bro project was maintaining bro plugins and it was > thought > > they might adopt this one. > > - Bro then formalized their plugin framework BUT dumped all plugins out > of > > their sphere of maintenance. > > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires > > that a package live within its own repo”. Jon said “the bro packages > model > > doesn't allow colocation with anything else.” > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo > > was created a few days later. But Metron wasn’t actually modified to > remove > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks > ago! > > – with https://issues.apache.org/jira/browse/METRON-1309 . > > - Presumably the need to have metron-bro-plugin-kafka in a separate repo > > remain valid, if the bro plugin mechanism is used. But obviously there > are > > (non-conforming) ways to build the plugin as part of metron, and install > it > > in a way that works. > > > > Q1. I think that last statement needs some explanation. Nick or Jon, can > > you please expand on it, especially wrt how the end user installs the > > plugin once the plugin is built the two different ways? And whether it’s > > still valuable to have a separate repo for the plugin? > > > > Nick suggests using a submodule approach to managing the bro plugin, for > > Metron versioning purposes. As I understand it, this would continue the > > existence of the metron-bro-plugin-kafka repo, but copy it into the > metron > > code tree for building, versioning, and release purposes. Git submodules > > are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules > . > > We would use the submodule capability to clone the > metron-bro-plugin-kafka > > source code into a subdirectory of Metron at the time one clones the > metron > > repo. It would then be released with Metron as part of the source code > > release for a given version of Metron. Part of the way submodules are > > managed, is that git stores the SHA1 hash of the submodule into a file > > named .gitmodules, which in turn gets saved when you do a git push. So > > indeed submodules would ensure that everyone cloning a given version of > > metron would get the expected “version” (sha, actually) of > > metron-bro-plugin-kafka. > > > > This sounds like a good idea, although it isn’t without cost. Submodules > > impose the need for additional commands to actually get a copy of the > > submodule source, and if the plugin repo advanced beyond the version in a > > metron repo, it causes some ‘git status’ artifacts that could be > confusing > > to folks who aren’t familiar with submodules. But these can be > documented. > > > > Q2. Nick, what I’m not clear about is the process by which the > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron > > developers, and (b) end users. If it “must” be in a separate repo to be > > successfully built and managed by the bro plugin mechanism, does that > mean > > it can’t be built from the copy in the Metron source tree? Yet until > > November, that’s exactly what we were doing. Do we go back to doing that? > > What does that mean wrt users installing the plugin? > > > > Thanks for your patience in reading this far. > > --Matt > > > > > > On 11/27/17, 2:58 PM, "James Sirota" <jsir...@apache.org> wrote: > > > > I agree with Nick. Since the plugin is tightly coupled with Metron why > not > > just pull it into the main repo and version it with the rest of the code? > > Do we really need the second repo for the plug-in? > > > > Thanks, > > James > > > > > > > > 16.11.2017, 08:06, "Nick Allen" <n...@nickallen.org>: > > >> I would suggest that we institute a release procedure for the package > > >> itself, but I don't think it necessarily has to line up with metron > > >> releases (happy to be persuaded otherwise). Then we can just link > metron > > >> to metron-bro-plugin-kafka by pointing to specific > > >> metron-bro-plugin-kafka releases (git tags > > >> < > > http://bro-package-manager.readthedocs.io/en/stable/ > package.html#package- > > >> versioning> > > >> ). > > >> Right now, full-dev spins up against the > > >> apache/metron-bro-plugin-kafka master branch, which is not a good idea > > to > > >> have in place for an upcoming release. That is the crux of why I think > > we > > >> need to finalize the move to bro 2.5.2 and the plugin packaging before > > our > > >> next release (working on it as we speak). > > >> Jon > > > > > > I replayed Jon's comments from the other thread above. > > > > > > My initial thought, is that I would not want to manage two separate > > release > > > processes. I don't want to have a roll call, cut release candidates and > > > test both. > > > > > > I was thinking we would just need to change some of the > behind-the-scenes > > > processes handled by the release manager. This is one area where I had > > > thought using a submodule in Git would help. > > > > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <n...@nickallen.org> > wrote: > > > > > >> + Restarting the thread to include mentors. > > >> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the > external > > >> repository that we set up a while back. > > >> > > >> - Metron Core: git://git.apache.org/metron.git > > >> - Kafka Plugin for Bro: git://git.apache.org/ > > >> metron-bro-plugin-kafka.git > > >> > > >> (Q) Do we need to change anything in the release procedure to account > > for > > >> this? > > > > ------------------- > > Thank you, > > > > James Sirota > > PMC- Apache Metron > > jsirota AT apache DOT org > > > > -- > > > > Jon > > > -- > > Jon >
