In an attempt to keep this from becoming unbearably long, I will try to keep my responses short, but I would be happy to elaborate. That's a fairly good timeline and summary, but here are some clarifications in corresponding order:
- The plugin history is quite short and you can probably get a good bit of context just by looking at the commits <https://github.com/apache/metron-bro-plugin-kafka/commits/master>. - The plugin is only useful to the bro community, but it is rather popular. - The Bro team created the idea of bro packages, which can include bro plugins, bro scripts, or BroControl plugins. So, instead of having a 'plugins' repo, they moved to have a 'packages' repo which is by default referenced by a bro-pkg tool they wrote for package management. - I believe I kicked this off (or at least I did in my head) when I started complaining <https://issues.apache.org/jira/browse/METRON-348?filter=-2> about the plugin divergence that occurred due to the move to bro/plugins (the right move at the time), but Metron's use of a local directory that hadn't been kept up to date. My current efforts are an attempt to make sure this doesn't happen again, and to take advantage of the bro-pkg benefits. - The gap between ~3/31 and actual progress on 11/12 is completely on me - I had intentions of doing this work sooner but failed to do so. - You can most definitely still install/use the bro plugin without using bro-pkg. In fact, the README in my PR <https://github.com/JonZeolla/metron-bro-plugin-kafka/blob/7402a48cab8be7802fa0c395648597ea6a1cbd71/README.md> still has the instructions on how to do so. Q1: The simple explanation is that the only thing that makes a plugin a bro package is the inclusion of a bro-pkg.meta file <https://github.com/apache/metron-bro-plugin-kafka/blob/master/bro-pkg.meta>, and it includes a build_command which could easily be manually performed to install by hand (assuming dependencies are met). I've worked with other projects that use submodules and while I'm fine discussing it, I suggest that we don't implement it. I put together a quick example of why here[1], using the bro project as an example since it's top of mind. Q2: I think the answer to Q1 answers this. There is absolutely nothing stopping a git clone && cd $dir && configure && make && make install, but using bro-pkg to install/load takes into account dependencies <https://github.com/apache/metron-bro-plugin-kafka/blob/master/bro-pkg.meta#L10-L14> and unit tests <https://github.com/apache/metron-bro-plugin-kafka/blob/master/bro-pkg.meta#L6> when it is loaded (and thus fails early and more intuitively). It only must be a separate repo (or, more technically correct, a git branch that includes only the package) because of how bro-pkg works. If you'd like to get an idea of how this would work in application for Bro users, you can see my test instructions here <https://github.com/apache/metron-bro-plugin-kafka/pull/4> (specifically step #3). If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install apache/metron-bro-plugin-kafka --version 0.1` due to this <https://github.com/bro/packages/blob/master/apache/bro-pkg.index> (the --force is just to remove user interaction, for an ansible spin-up). 1: To clone the Bro git repo, you must run `git clone --recursive https://github.com/bro/bro` (note the --recursive). Not too big of a deal, but requires that you remember it and existing instructions/blog posts may give users inaccurate steps. Let's make this worse and try to checkout their latest release, v2.5.2, and automatically update the submodules appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails because aux/plugins (https://github.com/bro/plugins) was removed since their latest release. Okay, we can work around this using `git checkout v2.5.2` and then remember to `git submodule update` every time you checkout a release or branch. But because they have nested submodules, we actually need to run `git submodule update --recursive`. I can't imagine opting *into* a workflow anything like this. There are other options as well, such as git subtrees, but those I am less familiar with. Jon On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ottobackwa...@gmail.com> wrote: > I am not sure that our use of the plugin necessarily equates to it being > implicitly coupled to Metron. It seems like the Right Thing To Do[image: > ™], esp. > for an Apache project would be to make this available for use by the > greater bro community. > Unless we expect to do extensive iterative work on the plugin, which would > then make the decision to spin it out now premature. > > Then again, I might be wrong ;) > > > On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote: > > [Please pardon me that the below is a little labored. I’m trying to > understand the implications for both release and use, which requires some > explanation as well as the two questions needed. Q1 and Q2 below are > probably the same question, asked in slightly different contexts. Please > consider them together.] > > So this made me go back and look at the history that caused us to put the > bro plugin in a separate repo. As best I can see, this was in > https://issues.apache.org/jira/browse/METRON-813 , which cites an email > discussion thread. Also please see > https://issues.apache.org/jira/browse/METRON-883 for background on the > plugin itself. > > As best I can assemble the many bits brought up in the threads, the reasons > to put it in a separate repo was: > - The plugin was thought to be useful to multiple clients of bro and kafka, > including Storm and Spark, as well as Metron. > - Originally the bro project was maintaining bro plugins and it was thought > they might adopt this one. > - Bro then formalized their plugin framework BUT dumped all plugins out of > their sphere of maintenance. > - As of 3/31/2017, Nick said that “the [bro] package mechanism requires > that a package live within its own repo”. Jon said “the bro packages model > doesn't allow colocation with anything else.” > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo > was created a few days later. But Metron wasn’t actually modified to remove > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago! > – with https://issues.apache.org/jira/browse/METRON-1309 . > - Presumably the need to have metron-bro-plugin-kafka in a separate repo > remain valid, if the bro plugin mechanism is used. But obviously there are > (non-conforming) ways to build the plugin as part of metron, and install it > in a way that works. > > Q1. I think that last statement needs some explanation. Nick or Jon, can > you please expand on it, especially wrt how the end user installs the > plugin once the plugin is built the two different ways? And whether it’s > still valuable to have a separate repo for the plugin? > > Nick suggests using a submodule approach to managing the bro plugin, for > Metron versioning purposes. As I understand it, this would continue the > existence of the metron-bro-plugin-kafka repo, but copy it into the metron > code tree for building, versioning, and release purposes. Git submodules > are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules . > We would use the submodule capability to clone the metron-bro-plugin-kafka > source code into a subdirectory of Metron at the time one clones the metron > repo. It would then be released with Metron as part of the source code > release for a given version of Metron. Part of the way submodules are > managed, is that git stores the SHA1 hash of the submodule into a file > named .gitmodules, which in turn gets saved when you do a git push. So > indeed submodules would ensure that everyone cloning a given version of > metron would get the expected “version” (sha, actually) of > metron-bro-plugin-kafka. > > This sounds like a good idea, although it isn’t without cost. Submodules > impose the need for additional commands to actually get a copy of the > submodule source, and if the plugin repo advanced beyond the version in a > metron repo, it causes some ‘git status’ artifacts that could be confusing > to folks who aren’t familiar with submodules. But these can be documented. > > Q2. Nick, what I’m not clear about is the process by which the > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron > developers, and (b) end users. If it “must” be in a separate repo to be > successfully built and managed by the bro plugin mechanism, does that mean > it can’t be built from the copy in the Metron source tree? Yet until > November, that’s exactly what we were doing. Do we go back to doing that? > What does that mean wrt users installing the plugin? > > Thanks for your patience in reading this far. > --Matt > > > On 11/27/17, 2:58 PM, "James Sirota" <jsir...@apache.org> wrote: > > I agree with Nick. Since the plugin is tightly coupled with Metron why not > just pull it into the main repo and version it with the rest of the code? > Do we really need the second repo for the plug-in? > > Thanks, > James > > > > 16.11.2017, 08:06, "Nick Allen" <n...@nickallen.org>: > >> I would suggest that we institute a release procedure for the package > >> itself, but I don't think it necessarily has to line up with metron > >> releases (happy to be persuaded otherwise). Then we can just link metron > >> to metron-bro-plugin-kafka by pointing to specific > >> metron-bro-plugin-kafka releases (git tags > >> < > http://bro-package-manager.readthedocs.io/en/stable/package.html#package- > >> versioning> > >> ). > >> Right now, full-dev spins up against the > >> apache/metron-bro-plugin-kafka master branch, which is not a good idea > to > >> have in place for an upcoming release. That is the crux of why I think > we > >> need to finalize the move to bro 2.5.2 and the plugin packaging before > our > >> next release (working on it as we speak). > >> Jon > > > > I replayed Jon's comments from the other thread above. > > > > My initial thought, is that I would not want to manage two separate > release > > processes. I don't want to have a roll call, cut release candidates and > > test both. > > > > I was thinking we would just need to change some of the behind-the-scenes > > processes handled by the release manager. This is one area where I had > > thought using a submodule in Git would help. > > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <n...@nickallen.org> wrote: > > > >> + Restarting the thread to include mentors. > >> > >> The code of the 'Kafka Plugin for Bro' is now maintained in the external > >> repository that we set up a while back. > >> > >> - Metron Core: git://git.apache.org/metron.git > >> - Kafka Plugin for Bro: git://git.apache.org/ > >> metron-bro-plugin-kafka.git > >> > >> (Q) Do we need to change anything in the release procedure to account > for > >> this? > > ------------------- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > -- Jon
