You and Matt should coordinate sending a mail to the dev list with a heads up, starting and done.
I think you mean that: Between XXXXX and YYYYYYY, if you do a fetch apache && checkout -b foo apache/master and then do vagrant up with the sensors enabled it will fail. Right? On December 7, 2017 at 15:09:52, zeo...@gmail.com (zeo...@gmail.com) wrote: FYI to be uber clear about the effects of what I'm doing, spinning up full-dev only when including the sensors will be broken on the bro plugin install step between when I push the changes, and when mattf pushes the 0.1 tag to apache/metron-bro-plugin-kafka. Jon On Thu, Dec 7, 2017 at 3:05 PM zeo...@gmail.com <zeo...@gmail.com> wrote: > Sounds good. Yes Matt, I will handle my parts now. Thanks everyone > > Jon > > On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <ma...@apache.org> wrote: > >> I can start the release process tonight. >> >> >> >> Jon, you mentioned you want to commit >> >> > https://github.com/apache/metron/pull/847 and >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4 >> >> before the release. Is it convenient for you to do so today? >> >> >> >> Thanks, >> >> --Matt >> >> >> >> From: Nick Allen <n...@nickallen.org> >> Date: Thursday, December 7, 2017 at 10:13 AM >> To: "dev@metron.apache.org" <dev@metron.apache.org> >> Cc: Matt Foley <ma...@apache.org> >> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro' >> >> >> >> I am more interested in getting a release cut. If me moving to the (a) >> camp gets us to consensus and cuts a release faster, then I'll do it. >> Let's get this release train moving. >> >> >> >> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <justinjl...@gmail.com> >> wrote: >> >> Do we have any further discussion on this? Pardon me if I misstate >> anyone's position, but it seems like we have a couple people (Otto and Jon >> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably >> a >> section of people like myself without a particular horse in the race. >> >> It seems like we need to come to some sort of consensus so that we can get >> the release bus moving again, and right now it seems like (a) is gathering >> more explicit support. Do we have a compelling reason to not do (a)? To >> be >> honest, my main worry is more "If we do (a) are we going to be miserable >> if >> we need to iterate or adjust?" I'm not seeing anything that suggests >> anything too terrible, so unless we see some more discussion, I suggest we >> move forward with (a). >> >> >> On Mon, Dec 4, 2017 at 9:34 PM, zeo...@gmail.com <zeo...@gmail.com> >> wrote: >> >> > I would prefer a, but I was initially thinking of doing the plugin first >> > and then get in the two PRs out to use this new tag, which are already >> +1'd >> > and just waiting on this conversation. For reference, >> > https://github.com/apache/metron/pull/847 and >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4 >> > >> > Jon >> > >> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ottobackwa...@gmail.com> wrote: >> > >> > > It seems to me, as I believe I have stated before that a) feels like >> the >> > > proper way to handle this. It is how I have seen other projects like >> > NiFi >> > > handle things as well. >> > > >> > > >> > > >> > > On December 4, 2017 at 17:14:41, Matt Foley (ma...@apache.org) wrote: >> > > >> > > Okay, looking at this from the perspective of making a release: >> > > >> > > >> > > >> > > We have two choices: >> > > >> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of >> > > metron-bro-plugin-kafka, at the same time and using the same process >> > > (modulo the necessary) as Metron. This is dirt simple. >> > > >> > > b) I or someone needs to: >> > > >> > > - open a jira, >> > > >> > > - add the submodule to the Metron code tree, >> > > >> > > - possibly (optionally) add build mechanism to the maven poms, and >> > > >> > > - document as much as we think appropriate regarding what it is, >> how >> > to >> > > build it, and how to update it, >> > > >> > > and commit that before the 0.4.2 release. >> > > >> > > >> > > >> > > What is the will of the community? >> > > >> > > Thanks, >> > > >> > > --Matt >> > > >> > > >> > > >> > > From: Nick Allen <n...@nickallen.org> >> > > Reply-To: "dev@metron.apache.org" <dev@metron.apache.org> >> > > Date: Tuesday, November 28, 2017 at 9:09 AM >> > > To: "dev@metron.apache.org" <dev@metron.apache.org> >> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for >> > Bro' >> > > >> > > >> > > >> > > I'll add a bit to Jon's technical comments. >> > > >> > > >> > > >> > > * We only created a separate repo because it was a technical >> requirement >> > to >> > > leverage the bro-pkg mechanism. >> > > >> > > * Leveraging the new bro-pkg mechanism has many advantages as >> outlined by >> > > Jon. >> > > >> > > * Enabling the bro-pkg mechanism is backwards compatible. I can >> install >> > the >> > > plugin exactly how we use to. >> > > >> > > >> > > >> > > While I agree with Jon's technical comments, I disagree with the >> > > non-technical ones. >> > > >> > > >> > > >> > > (1) I do not want to change our release management process. While we >> > needed >> > > to make a new repo (a technical change), I did not want that to change >> > how >> > > we operate as a community (our procedures, policies, versioning and >> > release >> > > cycles). >> > > >> > > >> > > >> > > (2) I see no value in adopting a separate release management process >> for >> > > the Bro plugin alone. Having a separate release process does not make >> the >> > > plugin *more* available to the Bro community. >> > > >> > > >> > > >> > > (3) I also see no value in positioning the plugin to be spun-out of >> the >> > > Metron project. It is a part of Metron and I want to see it benefit >> and >> > > evolve "the Apache-way". >> > > >> > > >> > > >> > > In my mind, the best way to accommodate the additional repo, while >> > > minimizing changes to our release management process, is to treat the >> new >> > > repo as a submodule. I fail to see significant downsides to this >> > approach. >> > > A few extract command-line options do not seem overly onerous to me. >> > > >> > > >> > > >> > > Many thanks go to Jon for all the hard work he has put in enhancing >> the >> > > plugin. >> > > >> > > >> > > >> > > >> > > >> > > On Mon, Nov 27, 2017 at 10:07 PM, zeo...@gmail.com <zeo...@gmail.com> >> > > wrote: >> > > >> > > In an attempt to keep this from becoming unbearably long, I will try >> to >> > > keep my responses short, but I would be happy to elaborate. That's a >> > fairly >> > > good timeline and summary, but here are some clarifications in >> > > corresponding order: >> > > >> > > >> > > >> > > - The plugin history is quite short and you can probably get a good >> bit >> > of >> > > context just by looking at the commits. >> > > >> > > - The plugin is only useful to the bro community, but it is rather >> > popular. >> > > >> > > - The Bro team created the idea of bro packages, which can include bro >> > > plugins, bro scripts, or BroControl plugins. So, instead of having a >> > > 'plugins' repo, they moved to have a 'packages' repo which is by >> default >> > > referenced by a bro-pkg tool they wrote for package management. >> > > >> > > - I believe I kicked this off (or at least I did in my head) when I >> > started >> > > complaining about the plugin divergence that occurred due to the move >> to >> > > bro/plugins (the right move at the time), but Metron's use of a local >> > > directory that hadn't been kept up to date. My current efforts are an >> > > attempt to make sure this doesn't happen again, and to take advantage >> of >> > > the bro-pkg benefits. >> > > >> > > - The gap between ~3/31 and actual progress on 11/12 is completely on >> me >> > - >> > > I had intentions of doing this work sooner but failed to do so. >> > > >> > > - You can most definitely still install/use the bro plugin without >> using >> > > bro-pkg. In fact, the README in my PR still has the instructions on >> how >> > to >> > > do so. >> > > >> > > >> > > >> > > Q1: The simple explanation is that the only thing that makes a plugin >> a >> > bro >> > > package is the inclusion of a bro-pkg.meta file, and it includes a >> > > build_command which could easily be manually performed to install by >> hand >> > > (assuming dependencies are met). >> > > >> > > >> > > >> > > I've worked with other projects that use submodules and while I'm fine >> > > discussing it, I suggest that we don't implement it. I put together a >> > quick >> > > example of why here[1], using the bro project as an example since it's >> > top >> > > of mind. >> > > >> > > >> > > >> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing >> > > stopping a git clone && cd $dir && configure && make && make install, >> but >> > > using bro-pkg to install/load takes into account dependencies and unit >> > > tests when it is loaded (and thus fails early and more intuitively). >> It >> > > only must be a separate repo (or, more technically correct, a git >> branch >> > > that includes only the package) because of how bro-pkg works. If you'd >> > like >> > > to get an idea of how this would work in application for Bro users, >> you >> > can >> > > see my test instructions here (specifically step #3). If a 0.1 tag >> gets >> > > pushed to apache/metron-bro-plugin-kafka, the command could be >> `bro-pkg >> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install >> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the >> --force is >> > > just to remove user interaction, for an ansible spin-up). >> > > >> > > >> > > >> > > >> > > >> > > 1: To clone the Bro git repo, you must run `git clone --recursive >> > > https://github.com/bro/bro` <https://github.com/bro/bro> < >> https://github.com/bro/bro> (note the >> > > --recursive). Not too big of a deal, >> > > but requires that you remember it and existing instructions/blog posts >> > may >> > > give users inaccurate steps. Let's make this worse and try to checkout >> > > their latest release, v2.5.2, and automatically update the submodules >> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This >> fails >> > > because aux/plugins (https://github.com/bro/plugins) was removed >> since >> > > their latest release. Okay, we can work around this using `git >> checkout >> > > v2.5.2` and then remember to `git submodule update` every time you >> > checkout >> > > a release or branch. But because they have nested submodules, we >> actually >> > > need to run `git submodule update --recursive`. I can't imagine opting >> > into >> > > a workflow anything like this. There are other options as well, such >> as >> > git >> > > subtrees, but those I am less familiar with. >> > > >> > > >> > > >> > > Jon >> > > >> > > >> > > >> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ottobackwa...@gmail.com> >> > > wrote: >> > > >> > > I am not sure that our use of the plugin necessarily equates to it >> being >> > > implicitly coupled to Metron. It seems like the Right Thing To Do, >> esp. >> > > for an Apache project would be to make this available for use by the >> > > greater bro community. >> > > Unless we expect to do extensive iterative work on the plugin, which >> > would >> > > then make the decision to spin it out now premature. >> > > >> > > Then again, I might be wrong ;) >> > > >> > > >> > > On November 27, 2017 at 19:58:11, Matt Foley (ma...@apache.org) >> wrote: >> > > >> > > [Please pardon me that the below is a little labored. I’m trying to >> > > understand the implications for both release and use, which requires >> some >> > > explanation as well as the two questions needed. Q1 and Q2 below are >> > > probably the same question, asked in slightly different contexts. >> Please >> > > consider them together.] >> > > >> > > So this made me go back and look at the history that caused us to put >> the >> > > bro plugin in a separate repo. As best I can see, this was in >> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an >> email >> > > discussion thread. Also please see >> > > https://issues.apache.org/jira/browse/METRON-883 for background on >> the >> > > plugin itself. >> > > >> > > As best I can assemble the many bits brought up in the threads, the >> > reasons >> > > to put it in a separate repo was: >> > > - The plugin was thought to be useful to multiple clients of bro and >> > kafka, >> > > including Storm and Spark, as well as Metron. >> > > - Originally the bro project was maintaining bro plugins and it was >> > thought >> > > they might adopt this one. >> > > - Bro then formalized their plugin framework BUT dumped all plugins >> out >> > of >> > > their sphere of maintenance. >> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism >> requires >> > > that a package live within its own repo”. Jon said “the bro packages >> > model >> > > doesn't allow colocation with anything else.” >> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka >> repo >> > > was created a few days later. But Metron wasn’t actually modified to >> > remove >> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the >> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks >> > ago! >> > > – with https://issues.apache.org/jira/browse/METRON-1309 . >> > > - Presumably the need to have metron-bro-plugin-kafka in a separate >> repo >> > > remain valid, if the bro plugin mechanism is used. But obviously there >> > are >> > > (non-conforming) ways to build the plugin as part of metron, and >> install >> > it >> > > in a way that works. >> > > >> > > Q1. I think that last statement needs some explanation. Nick or Jon, >> can >> > > you please expand on it, especially wrt how the end user installs the >> > > plugin once the plugin is built the two different ways? And whether >> it’s >> > > still valuable to have a separate repo for the plugin? >> > > >> > > Nick suggests using a submodule approach to managing the bro plugin, >> for >> > > Metron versioning purposes. As I understand it, this would continue >> the >> > > existence of the metron-bro-plugin-kafka repo, but copy it into the >> > metron >> > > code tree for building, versioning, and release purposes. Git >> submodules >> > > are documented here: >> https://git-scm.com/book/en/v2/Git-Tools-Submodules >> > . >> > > We would use the submodule capability to clone the >> > metron-bro-plugin-kafka >> > > source code into a subdirectory of Metron at the time one clones the >> > metron >> > > repo. It would then be released with Metron as part of the source code >> > > release for a given version of Metron. Part of the way submodules are >> > > managed, is that git stores the SHA1 hash of the submodule into a file >> > > named .gitmodules, which in turn gets saved when you do a git push. So >> > > indeed submodules would ensure that everyone cloning a given version >> of >> > > metron would get the expected “version” (sha, actually) of >> > > metron-bro-plugin-kafka. >> > > >> > > This sounds like a good idea, although it isn’t without cost. >> Submodules >> > > impose the need for additional commands to actually get a copy of the >> > > submodule source, and if the plugin repo advanced beyond the version >> in a >> > > metron repo, it causes some ‘git status’ artifacts that could be >> > confusing >> > > to folks who aren’t familiar with submodules. But these can be >> > documented. >> > > >> > > Q2. Nick, what I’m not clear about is the process by which the >> > > metron-bro-plugin-kafka would be built and “plugged in” by (a) metron >> > > developers, and (b) end users. If it “must” be in a separate repo to >> be >> > > successfully built and managed by the bro plugin mechanism, does that >> > mean >> > > it can’t be built from the copy in the Metron source tree? Yet until >> > > November, that’s exactly what we were doing. Do we go back to doing >> that? >> > > What does that mean wrt users installing the plugin? >> > > >> > > Thanks for your patience in reading this far. >> > > --Matt >> > > >> > > >> > > On 11/27/17, 2:58 PM, "James Sirota" <jsir...@apache.org> wrote: >> > > >> > > I agree with Nick. Since the plugin is tightly coupled with Metron why >> > not >> > > just pull it into the main repo and version it with the rest of the >> code? >> > > Do we really need the second repo for the plug-in? >> > > >> > > Thanks, >> > > James >> > > >> > > >> > > >> > > 16.11.2017, 08:06, "Nick Allen" <n...@nickallen.org>: >> > > >> I would suggest that we institute a release procedure for the >> package >> > > >> itself, but I don't think it necessarily has to line up with metron >> > > >> releases (happy to be persuaded otherwise). Then we can just link >> > metron >> > > >> to metron-bro-plugin-kafka by pointing to specific >> > > >> metron-bro-plugin-kafka releases (git tags >> > > >> < >> > > http://bro-package-manager.readthedocs.io/en/stable/ >> > package.html#package- >> > > >> versioning> >> > > >> ). >> > > >> Right now, full-dev spins up against the >> > > >> apache/metron-bro-plugin-kafka master branch, which is not a good >> idea >> > > to >> > > >> have in place for an upcoming release. That is the crux of why I >> think >> > > we >> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging >> before >> > > our >> > > >> next release (working on it as we speak). >> > > >> Jon >> > > > >> > > > I replayed Jon's comments from the other thread above. >> > > > >> > > > My initial thought, is that I would not want to manage two separate >> > > release >> > > > processes. I don't want to have a roll call, cut release candidates >> and >> > > > test both. >> > > > >> > > > I was thinking we would just need to change some of the >> > behind-the-scenes >> > > > processes handled by the release manager. This is one area where I >> had >> > > > thought using a submodule in Git would help. >> > > > >> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <n...@nickallen.org> >> > wrote: >> > > > >> > > >> + Restarting the thread to include mentors. >> > > >> >> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in the >> > external >> > > >> repository that we set up a while back. >> > > >> >> > > >> - Metron Core: git://git.apache.org/metron.git >> > > >> - Kafka Plugin for Bro: git://git.apache.org/ >> > > >> metron-bro-plugin-kafka.git >> > > >> >> > > >> (Q) Do we need to change anything in the release procedure to >> account >> > > for >> > > >> this? >> > > >> > > ------------------- >> > > Thank you, >> > > >> > > James Sirota >> > > PMC- Apache Metron >> > > jsirota AT apache DOT org >> > > >> > > -- >> > > >> > > Jon >> > > >> > -- >> > >> > Jon >> > >> >> >> >> -- > > Jon > -- Jon
