It seems to me, as I believe I have stated before that a) feels like the
proper way to handle this. It is how I have seen other projects like NiFi
handle things as well.
On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote:
Okay, looking at this from the perspective of making a release:
We have two choices:
a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
metron-bro-plugin-kafka, at the same time and using the same process
(modulo the necessary) as Metron. This is dirt simple.
b) I or someone needs to:
- open a jira,
- add the submodule to the Metron code tree,
- possibly (optionally) add build mechanism to the maven poms, and
- document as much as we think appropriate regarding what it is, how to
build it, and how to update it,
and commit that before the 0.4.2 release.
What is the will of the community?
Thanks,
--Matt
From: Nick Allen <n...@nickallen.org>
Reply-To: "dev@metron.apache.org" <dev@metron.apache.org>
Date: Tuesday, November 28, 2017 at 9:09 AM
To: "dev@metron.apache.org" <dev@metron.apache.org>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
I'll add a bit to Jon's technical comments.
* We only created a separate repo because it was a technical requirement to
leverage the bro-pkg mechanism.
* Leveraging the new bro-pkg mechanism has many advantages as outlined by
Jon.
* Enabling the bro-pkg mechanism is backwards compatible. I can install the
plugin exactly how we use to.
While I agree with Jon's technical comments, I disagree with the
non-technical ones.
(1) I do not want to change our release management process. While we needed
to make a new repo (a technical change), I did not want that to change how
we operate as a community (our procedures, policies, versioning and release
cycles).
(2) I see no value in adopting a separate release management process for
the Bro plugin alone. Having a separate release process does not make the
plugin *more* available to the Bro community.
(3) I also see no value in positioning the plugin to be spun-out of the
Metron project. It is a part of Metron and I want to see it benefit and
evolve "the Apache-way".
In my mind, the best way to accommodate the additional repo, while
minimizing changes to our release management process, is to treat the new
repo as a submodule. I fail to see significant downsides to this approach.
A few extract command-line options do not seem overly onerous to me.
Many thanks go to Jon for all the hard work he has put in enhancing the
plugin.
On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com <zeo...@gmail.com>
wrote:
In an attempt to keep this from becoming unbearably long, I will try to
keep my responses short, but I would be happy to elaborate. That's a fairly
good timeline and summary, but here are some clarifications in
corresponding order:
- The plugin history is quite short and you can probably get a good bit of
context just by looking at the commits.
- The plugin is only useful to the bro community, but it is rather popular.
- The Bro team created the idea of bro packages, which can include bro
plugins, bro scripts, or BroControl plugins. So, instead of having a
'plugins' repo, they moved to have a 'packages' repo which is by default
referenced by a bro-pkg tool they wrote for package management.
- I believe I kicked this off (or at least I did in my head) when I started
complaining about the plugin divergence that occurred due to the move to
bro/plugins (the right move at the time), but Metron's use of a local
directory that hadn't been kept up to date. My current efforts are an
attempt to make sure this doesn't happen again, and to take advantage of
the bro-pkg benefits.
- The gap between ~3/31 and actual progress on 11/12 is completely on me -
I had intentions of doing this work sooner but failed to do so.
- You can most definitely still install/use the bro plugin without using
bro-pkg. In fact, the README in my PR still has the instructions on how to
do so.
Q1: The simple explanation is that the only thing that makes a plugin a bro
package is the inclusion of a bro-pkg.meta file, and it includes a
build_command which could easily be manually performed to install by hand
(assuming dependencies are met).
I've worked with other projects that use submodules and while I'm fine
discussing it, I suggest that we don't implement it. I put together a quick
example of why here[1], using the bro project as an example since it's top
of mind.
Q2: I think the answer to Q1 answers this. There is absolutely nothing
stopping a git clone && cd $dir && configure && make && make install, but
using bro-pkg to install/load takes into account dependencies and unit
tests when it is loaded (and thus fails early and more intuitively). It
only must be a separate repo (or, more technically correct, a git branch
that includes only the package) because of how bro-pkg works. If you'd like
to get an idea of how this would work in application for Bro users, you can
see my test instructions here (specifically step #3). If a 0.1 tag gets
pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
just to remove user interaction, for an ansible spin-up).
1: To clone the Bro git repo, you must run `git clone --recursive
https://github.com/bro/bro` (note the --recursive). Not too big of a deal,
but requires that you remember it and existing instructions/blog posts may
give users inaccurate steps. Let's make this worse and try to checkout
their latest release, v2.5.2, and automatically update the submodules
appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
because aux/plugins (https://github.com/bro/plugins) was removed since
their latest release. Okay, we can work around this using `git checkout
v2.5.2` and then remember to `git submodule update` every time you checkout
a release or branch. But because they have nested submodules, we actually
need to run `git submodule update --recursive`. I can't imagine opting into
a workflow anything like this. There are other options as well, such as git
subtrees, but those I am less familiar with.
Jon
On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ottobackwa...@gmail.com>
wrote:
I am not sure that our use of the plugin necessarily equates to it being
implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
for an Apache project would be to make this available for use by the
greater bro community.
Unless we expect to do extensive iterative work on the plugin, which would
then make the decision to spin it out now premature.
Then again, I might be wrong ;)
On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) wrote:
[Please pardon me that the below is a little labored. I’m trying to
understand the implications for both release and use, which requires some
explanation as well as the two questions needed. Q1 and Q2 below are
probably the same question, asked in slightly different contexts. Please
consider them together.]
So this made me go back and look at the history that caused us to put the
bro plugin in a separate repo. As best I can see, this was in
https://issues.apache.org/jira/browse/METRON-813 , which cites an email
discussion thread. Also please see
https://issues.apache.org/jira/browse/METRON-883 for background on the
plugin itself.
As best I can assemble the many bits brought up in the threads, the reasons
to put it in a separate repo was:
- The plugin was thought to be useful to multiple clients of bro and kafka,
including Storm and Spark, as well as Metron.
- Originally the bro project was maintaining bro plugins and it was thought
they might adopt this one.
- Bro then formalized their plugin framework BUT dumped all plugins out of
their sphere of maintenance.
- As of 3/31/2017, Nick said that “the [bro] package mechanism requires
that a package live within its own repo”. Jon said “the bro packages model
doesn't allow colocation with anything else.”
- So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
was created a few days later. But Metron wasn’t actually modified to remove
the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
– with https://issues.apache.org/jira/browse/METRON-1309 .
- Presumably the need to have metron-bro-plugin-kafka in a separate repo
remain valid, if the bro plugin mechanism is used. But obviously there are
(non-conforming) ways to build the plugin as part of metron, and install it
in a way that works.
Q1. I think that last statement needs some explanation. Nick or Jon, can
you please expand on it, especially wrt how the end user installs the
plugin once the plugin is built the two different ways? And whether it’s
still valuable to have a separate repo for the plugin?
Nick suggests using a submodule approach to managing the bro plugin, for
Metron versioning purposes. As I understand it, this would continue the
existence of the metron-bro-plugin-kafka repo, but copy it into the metron
code tree for building, versioning, and release purposes. Git submodules
are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
We would use the submodule capability to clone the metron-bro-plugin-kafka
source code into a subdirectory of Metron at the time one clones the metron
repo. It would then be released with Metron as part of the source code
release for a given version of Metron. Part of the way submodules are
managed, is that git stores the SHA1 hash of the submodule into a file
named .gitmodules, which in turn gets saved when you do a git push. So
indeed submodules would ensure that everyone cloning a given version of
metron would get the expected “version” (sha, actually) of
metron-bro-plugin-kafka.
This sounds like a good idea, although it isn’t without cost. Submodules
impose the need for additional commands to actually get a copy of the
submodule source, and if the plugin repo advanced beyond the version in a
metron repo, it causes some ‘git status’ artifacts that could be confusing
to folks who aren’t familiar with submodules. But these can be documented.
Q2. Nick, what I’m not clear about is the process by which the
metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
developers, and (b) end users. If it “must” be in a separate repo to be
successfully built and managed by the bro plugin mechanism, does that mean
it can’t be built from the copy in the Metron source tree? Yet until
November, that’s exactly what we were doing. Do we go back to doing that?
What does that mean wrt users installing the plugin?
Thanks for your patience in reading this far.
--Matt
On 11/27/17, 2:58 PM, "James Sirota" <jsir...@apache.org> wrote:
I agree with Nick. Since the plugin is tightly coupled with Metron why not
just pull it into the main repo and version it with the rest of the code?
Do we really need the second repo for the plug-in?
Thanks,
James
16.11.2017, 08:06, "Nick Allen" <n...@nickallen.org>:
>> I would suggest that we institute a release procedure for the package
>> itself, but I don't think it necessarily has to line up with metron
>> releases (happy to be persuaded otherwise). Then we can just link metron
>> to metron-bro-plugin-kafka by pointing to specific
>> metron-bro-plugin-kafka releases (git tags
>> <
http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
>> versioning>
>> ).
>> Right now, full-dev spins up against the
>> apache/metron-bro-plugin-kafka master branch, which is not a good idea
to
>> have in place for an upcoming release. That is the crux of why I think
we
>> need to finalize the move to bro 2.5.2 and the plugin packaging before
our
>> next release (working on it as we speak).
>> Jon
>
> I replayed Jon's comments from the other thread above.
>
> My initial thought, is that I would not want to manage two separate
release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <n...@nickallen.org> wrote:
>
>> + Restarting the thread to include mentors.
>>
>> The code of the 'Kafka Plugin for Bro' is now maintained in the external
>> repository that we set up a while back.
>>
>> - Metron Core: git://git.apache.org/metron.git
>> - Kafka Plugin for Bro: git://git.apache.org/
>> metron-bro-plugin-kafka.git
>>
>> (Q) Do we need to change anything in the release procedure to account
for
>> this?
-------------------
Thank you,
James Sirota
PMC- Apache Metron
jsirota AT apache DOT org
--
Jon
