So here's an update on this, and I'm looking for any suggestions regarding the roadmap. If anybody isn't clear about the implementation specifics or history here, I'm happy to reiterate, but it has also been discussed on the mailing list in the past.
Right now, we have a direct migration of metron-sensors/bro-plugin-kafka[1] to its own repository[2], which is required in order to turn it into a bro package (the preferred mechanism of managing and installing bro plugins as of 2.5). Bro 2.5 also has an improvement[3] which I think we should consider important to take advantage of in our testing environments and instructions. Here is a more well-defined roadmap suggestion: 1. *DONE* - Move the bro-plugin-kafka to its own repository (Prereq to METRON-813). 2. *PR OPEN* - Reorganize the plugin naming to be more appropriate for a package (METRON-1303, Prereq to METRON-813). 3. Transform the plugin to be a package containing a plugin (METRON-813). 4. Add the package to bro/packages[4]. During 2-4. Upgrade full-dev to run on CentOS 7 (METRON-559, soft prereq to METRON-1088). 5. Upgrade bro to 2.5.2 (METRON-1088). 6. Update metron-deployment to install the package instead of using the plugin in apache/metron. 7. Remove the plugin[1] from apache/metron entirely. 8. Add features/improvements (METRON-1304, METRON-1305, etc.) to apache/metron-bro-plugin-kafka. - This is why my METRON-1304 PR has "DO NOT MERGE" - it was simply an easy win to bust out this morning while I was thinking of it. Some alternatives we may want to consider: - In the interim, we could change our ansible scripts to pull in the code from the external repository, but build it the same as we currently do (as a plugin but not a package). - We could replace bro-plugin-kafka folder with a sub-module pointing to the external repo, but this may be more trouble than it's worth (new cloning instructions, new folder name, etc.). - We can try to work around some of the issues with running Bro 2.5 on CentOS 6, but it would be less preferred. Any comments are welcome. 1: https://github.com/apache/metron/tree/master/metron-sensors/bro-plugin-kafka 2: https://github.com/apache/metron-bro-plugin-kafka 3: https://www.bro.org/sphinx/cluster/index.html#logger 4: https://github.com/bro/packages On Tue, Nov 7, 2017 at 8:52 AM Nick Allen <n...@nickallen.org> wrote: > > As of bro-pkg 1.1, I need to redo my `bro-pkg.meta` work to support the > newly-added > `external_depends`, and also upgrade to bro 2.5.2 > > Isn't upgrading to 2.5.2 an enhancement that we need to wait on before we > finish some clean-up tasks? > > What all do you think we need to do before we start accepting enhancements? > > Thanks for the update and all the hard work, Jon. > > On Mon, Nov 6, 2017 at 10:02 PM, zeo...@gmail.com <zeo...@gmail.com> > wrote: > > > Sorry for the delay here - I pushed this out tonight (link > > <https://github.com/apache/metron-bro-plugin-kafka/commits/master>, link > > <https://git-wip-us.apache.org/repos/asf?p=metron-bro-plugin-kafka.git > >). > > As of bro-pkg 1.1, I need to redo my `bro-pkg.meta` work to support the > > newly-added `external_depends`, and also upgrade to bro 2.5.2 (somewhat > > non-trivial due to the C++11 requirement, and new bro log files/fields) > so > > we can use the bro package manager to install the plugin. Hopefully I > can > > get this wrapped up soon so we can accept external PRs like this one > > <https://github.com/JonZeolla/metron-bro-plugin-kafka/pull/1>. > > > > Jon > > > > On Mon, Sep 18, 2017 at 11:52 AM Nick Allen <n...@nickallen.org> wrote: > > > > > Nice! Looks good to me. > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 18, 2017 at 11:35 AM zeo...@gmail.com <zeo...@gmail.com> > > > wrote: > > > > > > > Okay, I took a stab at it this morning, can I get a double check > before > > > > pushing it out? The latest commit would be opened as a PR. > > > > > > > > https://github.com/JonZeolla/metron-bro-plugin-kafka/tree/dev > > > > > > > > Jon > > > > > > > > On Fri, Sep 15, 2017 at 12:54 PM zeo...@gmail.com <zeo...@gmail.com> > > > > wrote: > > > > > > > > > Good point, I can take that task re migrating the revision history > of > > > the > > > > > folder. > > > > > > > > > > Jon > > > > > > > > > > On Fri, Sep 15, 2017, 12:14 Nick Allen <n...@nickallen.org> wrote: > > > > > > > > > >> Hi Jon - > > > > >> > > > > >> I agree with you on the approach. We should first copy everything > > as > > > it > > > > >> is > > > > >> to the new repo. We should maintain the revision history too. > I'm > > > sure > > > > >> there is a way to do it, but would have to research a bit. Then > we > > > > apply > > > > >> your changes on top of that. > > > > >> > > > > >> Thanks > > > > >> > > > > >> On Thu, Sep 14, 2017 at 1:36 AM, zeo...@gmail.com < > zeo...@gmail.com > > > > > > > >> wrote: > > > > >> > > > > >> > So, I've been working on METRON-813 > > > > >> > <https://issues.apache.org/jira/browse/METRON-813> lately and I > > > have > > > > an > > > > >> > initial run at it ready to go here > > > > >> > <https://github.com/JonZeolla/metron-bro-plugin-kafka> > (squashed > > > > >> history, > > > > >> > see a better history there > > > > >> > < > > > https://github.com/JonZeolla/metron-bro-plugin-kafka/commits/bro-pkg > > > > >> >). > > > > >> > Since the metron-bro-plugin-kafka repo is empty, I can't open a > PR > > > > >> against > > > > >> > it on GitHub for review. Does anybody have a suggestion > regarding > > > how > > > > >> to > > > > >> > move forward? I see two options: > > > > >> > 1. I make the initial commit a direct copy of the > bro-plugin-kafka > > > > >> folder > > > > >> > <https://github.com/apache/metron/tree/master/metron- > > > > >> > sensors/bro-plugin-kafka> > > > > >> > (I believe this would require a new JIRA for a direct copy), and > > > then > > > > >> open > > > > >> > a PR for the METRON-813 changes to get reviewed via the normal > > > > process. > > > > >> > 2. I make the initial commit the result of METRON-813, but > review > > > > occurs > > > > >> > via the mailing list and using my fork. > > > > >> > > > > > >> > I prefer 1, but wanted to put it up for discussion. Once we > > decide > > > on > > > > >> the > > > > >> > correct approach then I would be happy to put together a testing > > > plan > > > > >> for > > > > >> > the PR as well. > > > > >> > > > > > >> > Just to clarify, the general roadmap for getting this used in > > > > >> apache/metron > > > > >> > is: > > > > >> > 1. Create a bro package in apache/metron-bro-plugin-kafka > > > > >> > 2. Update the ansible bro setup > > > > >> > <https://github.com/apache/metron/tree/master/metron- > > > > >> > deployment/roles/bro/tasks> > > > > >> > to install/configure bro-pkg (`pip install bro-pkg && bro-pkg > > > > >> autoconfig`) > > > > >> > and use it to install the apache/metron-bro-plugin-kafka > package. > > > > >> > > > > > >> > I will also be adding this to the official bro package manager > > > > >> > <https://github.com/bro/packages>, but out of an abundance of > > > > caution I > > > > >> > plan to setup ansible to pull the package directly from the > > > > >> > apache/metron-bro-plugin-kafka using bro-pkg instead of going > > > through > > > > >> the > > > > >> > bro/packages package source (which removes the bro/packages > > > > dependency). > > > > >> > > > > > >> > Feedback on all of the above is welcome. > > > > >> > > > > > >> > Jon > > > > >> > -- > > > > >> > > > > > >> > Jon > > > > >> > > > > > >> > > > > > -- > > > > > > > > > > Jon > > > > > > > > > -- > > > > > > > > Jon > > > > > > > > > -- > > > > Jon > > > -- Jon
