I am not sure that our use of the plugin necessarily equates to it being implicitly coupled to Metron. It seems like the Right Thing To Do™, esp. for an Apache project would be to make this available for use by the greater bro community. Unless we expect to do extensive iterative work on the plugin, which would then make the decision to spin it out now premature.
Then again, I might be wrong ;) On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote: [Please pardon me that the below is a little labored. I’m trying to understand the implications for both release and use, which requires some explanation as well as the two questions needed. Q1 and Q2 below are probably the same question, asked in slightly different contexts. Please consider them together.] So this made me go back and look at the history that caused us to put the bro plugin in a separate repo. As best I can see, this was in https://issues.apache.org/jira/browse/METRON-813 , which cites an email discussion thread. Also please see https://issues.apache.org/jira/browse/METRON-883 for background on the plugin itself. As best I can assemble the many bits brought up in the threads, the reasons to put it in a separate repo was: - The plugin was thought to be useful to multiple clients of bro and kafka, including Storm and Spark, as well as Metron. - Originally the bro project was maintaining bro plugins and it was thought they might adopt this one. - Bro then formalized their plugin framework BUT dumped all plugins out of their sphere of maintenance. - As of 3/31/2017, Nick said that “the [bro] package mechanism requires that a package live within its own repo”. Jon said “the bro packages model doesn't allow colocation with anything else.” - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo was created a few days later. But Metron wasn’t actually modified to remove the metron-sensors/bro-plugin-kafka/ subdirectory and start using the plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago! – with https://issues.apache.org/jira/browse/METRON-1309 . - Presumably the need to have metron-bro-plugin-kafka in a separate repo remain valid, if the bro plugin mechanism is used. But obviously there are (non-conforming) ways to build the plugin as part of metron, and install it in a way that works. Q1. I think that last statement needs some explanation. Nick or Jon, can you please expand on it, especially wrt how the end user installs the plugin once the plugin is built the two different ways? And whether it’s still valuable to have a separate repo for the plugin? Nick suggests using a submodule approach to managing the bro plugin, for Metron versioning purposes. As I understand it, this would continue the existence of the metron-bro-plugin-kafka repo, but copy it into the metron code tree for building, versioning, and release purposes. Git submodules are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules . We would use the submodule capability to clone the metron-bro-plugin-kafka source code into a subdirectory of Metron at the time one clones the metron repo. It would then be released with Metron as part of the source code release for a given version of Metron. Part of the way submodules are managed, is that git stores the SHA1 hash of the submodule into a file named .gitmodules, which in turn gets saved when you do a git push. So indeed submodules would ensure that everyone cloning a given version of metron would get the expected “version” (sha, actually) of metron-bro-plugin-kafka. This sounds like a good idea, although it isn’t without cost. Submodules impose the need for additional commands to actually get a copy of the submodule source, and if the plugin repo advanced beyond the version in a metron repo, it causes some ‘git status’ artifacts that could be confusing to folks who aren’t familiar with submodules. But these can be documented. Q2. Nick, what I’m not clear about is the process by which the metron-bro-plugin-kafka would be built and “plugged in” by (a) metron developers, and (b) end users. If it “must” be in a separate repo to be successfully built and managed by the bro plugin mechanism, does that mean it can’t be built from the copy in the Metron source tree? Yet until November, that’s exactly what we were doing. Do we go back to doing that? What does that mean wrt users installing the plugin? Thanks for your patience in reading this far. --Matt On 11/27/17, 2:58 PM, "James Sirota" <jsir...@apache.org> wrote: I agree with Nick. Since the plugin is tightly coupled with Metron why not just pull it into the main repo and version it with the rest of the code? Do we really need the second repo for the plug-in? Thanks, James 16.11.2017, 08:06, "Nick Allen" <n...@nickallen.org>: >> I would suggest that we institute a release procedure for the package >> itself, but I don't think it necessarily has to line up with metron >> releases (happy to be persuaded otherwise). Then we can just link metron >> to metron-bro-plugin-kafka by pointing to specific >> metron-bro-plugin-kafka releases (git tags >> < http://bro-package-manager.readthedocs.io/en/stable/package.html#package- >> versioning> >> ). >> Right now, full-dev spins up against the >> apache/metron-bro-plugin-kafka master branch, which is not a good idea to >> have in place for an upcoming release. That is the crux of why I think we >> need to finalize the move to bro 2.5.2 and the plugin packaging before our >> next release (working on it as we speak). >> Jon > > I replayed Jon's comments from the other thread above. > > My initial thought, is that I would not want to manage two separate release > processes. I don't want to have a roll call, cut release candidates and > test both. > > I was thinking we would just need to change some of the behind-the-scenes > processes handled by the release manager. This is one area where I had > thought using a submodule in Git would help. > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <n...@nickallen.org> wrote: > >> + Restarting the thread to include mentors. >> >> The code of the 'Kafka Plugin for Bro' is now maintained in the external >> repository that we set up a while back. >> >> - Metron Core: git://git.apache.org/metron.git >> - Kafka Plugin for Bro: git://git.apache.org/ >> metron-bro-plugin-kafka.git >> >> (Q) Do we need to change anything in the release procedure to account for >> this? ------------------- Thank you, James Sirota PMC- Apache Metron jsirota AT apache DOT org
