[VOTE] Metron Release Candidate 0.7.1-RC2
This is a call to vote on releasing Apache Metron 0.7.1 Full list of changes in this release: https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/CHANGES The tag to be voted upon is: apache-metron_0.7.1-rc2 The source archives being voted upon can be found here: https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/apache-metron_0.7.1-rc2.tar.gz Other release files, signatures and digests can be found here: https://dist.apache.org/repos/dist/dev/metron/0.7.1-RC2/ The release artifacts are signed with the following key: https://dist.apache.org/repos/dist/release/metron/KEYS Please vote on releasing this package as Apache Metron 0.7.1-RC2 When voting, please list the actions taken to verify the release. Recommended build validation and verification instructions are posted here: https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds This vote will be open for until 7pm EDT on Monday May 13 2019, to account for the weekend. [ ] +1 Release this package as Apache Metron 0.7.1-RC2 [ ] 0 No opinion [ ] -1 Do not release this package because...
Re: [RESULT] [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images
Dev guidelines have been updated - https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines PR is up for the PR checklist change - https://issues.apache.org/jira/browse/METRON-2107 - https://github.com/apache/metron/pull/1401 On Wed, May 8, 2019 at 10:17 AM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Vote has passed. Voting was: > 2 binding +1's (Mike Miklavcic, Otto Fowler) > 1 non-binding +1 (Jon Zeolla) > > I will start making the changes to the dev guidelines today. > > Vote thread - > https://lists.apache.org/thread.html/dccecd41314e76eed7440ab8214ab5ee1ed28724e6f0e545c68ea5e0@%3Cdev.metron.apache.org%3E > >
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Sure, I can kick it off. I'd expect that to drop either tonight or tomorrow morning, depending on when I can dedicate a bit of time. Thanks to everyone who's helped (and continued to help) with working on this! On Wed, May 8, 2019 at 12:23 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > Hey everyone, > > METRON-2100 has been merged - https://github.com/apache/metron/pull/1398, > and the parser aggregation UI work and feature branch is under way. > > Justin, can we kick off an RC2? > > Cheers, > Mike > > On Fri, May 3, 2019 at 6:06 AM Otto Fowler > wrote: > > > Despite the name, we *have* been using it as both for quite some amount > of > > time. It *is* both dev and demo, and we recommend it as such on the list > > all the time. > > > > So there isn’t a decision to be made here as far as the status quo -> we > > use full dev as both dev and demo. > > > > > > > > > > On May 2, 2019 at 18:53:37, Michael Miklavcic ( > michael.miklav...@gmail.com > > ) > > wrote: > > > > Whether or not full dev is, first and foremost, "dev" I think your > > questions being up a good point. If not full_dev for introducing new > users, > > then what? If we want to provide a different env for letting people > tinker > > and try it out than we do for development, that's completely fine. But we > > don't have that right now. So we can treat full_dev as multipurpose, or > we > > can stop directing non-devs to it, or we can add something new. I > honestly > > don't have any recommendations here. We've talked about docker instances > > for replacing in-memory components, but I'm still not sure that solves > this > > problem, or adds more complexity. Given the current options on the table, > > I'm inclined to go with "full_dev" serves both dev and demo purposes. > Otto, > > what do you think? > > > > On Thu, May 2, 2019, 4:32 PM Otto Fowler > wrote: > > > > > I’ve commented on the PR, and I won’t repeat it here as well, I will > > > however ask again if we know and can list all of the usability issues > > that > > > surround this problem. IE. All the things that can happen or may happen > > > for people who are not Metron developers and committers who are using > > > full dev, because we keep recommending it. > > > > > > > > > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic ( > > michael.miklav...@gmail.com > > > ) > > > wrote: > > > > > > PR is up. I added the doc change to the metron-deployment README since > > this > > > serves as the gateway doc for all the VM instances. All of which would > be > > > affected by the feature gap. > > > > > > https://github.com/apache/metron/pull/1398 > > > > > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > > > Here's the ticket I created to track it, which also references the > Jira > > > > for the new UI feature. > > > > https://issues.apache.org/jira/browse/METRON-2100 > > > > > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > > > michael.miklav...@gmail.com> wrote: > > > > > > > >> :-) > > > >> > > > >> I expect to have #2 out sometime today. > > > >> > > > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > > > wrote: > > > >> > > > >>> > > > > >>> > I personally > > > >>> > don't like this feature gap in full dev. It seems Otto agrees, > and > > > >>> Casey at > > > >>> > the very least sees it as enough of an issue to gate us from 0.8. > > > >>> > > > > >>> > > > >>> +1 on all of this. I don't like it either. > > > >>> > > > >>> > > > >>> > Our vote landed 2-2. We are having a discussion about what to do > > with > > > >>> the > > > >>> > release. This is that discussion. > > > >>> > > > >>> > > > >>> I'm going to be honest, my response was a combination of misreading > > > what > > > >>> you said and thinking you were proposing delaying the release more > > > >>> seriously and feeling a bit blindsided by a perceived move from the > > > >>> initial > > > >>> "take more time than originally anticipated" (which in my head I > took > > > as > > > >>> a > > > >>> couple days) to "versus next week, or the week after" (where > delaying > > > >>> things weeks is something I personally would like not buried so far > > > down > > > >>> in > > > >>> the thread). Totally my bad, sorry about that. > > > >>> > > > >>> Other than that, it sounds like we're pretty much in agreement. > > > >>> > > > >>> Here's my current understanding of the state and consensus as of > > right > > > >>> now > > > >>> (which is subject to change as more discussion happens): > > > >>> > > > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and > > #3 > > > >>> for 0.8.0. > > > >>> - I don't believe I've seen an explicit response from Otto on what > > > >>> he > > > >>> thinks about doing this, and from a personal perspective like to > > > >>> see what > > > >>> his opinion is as the person who originally brought it up. > > > >>> - Mike said he's going to kick out a PR that addresses #2 > > > >>> - After that undergoes
Re: [DISCUSS] Metron Release - 0.7.1 next steps
Hey everyone, METRON-2100 has been merged - https://github.com/apache/metron/pull/1398, and the parser aggregation UI work and feature branch is under way. Justin, can we kick off an RC2? Cheers, Mike On Fri, May 3, 2019 at 6:06 AM Otto Fowler wrote: > Despite the name, we *have* been using it as both for quite some amount of > time. It *is* both dev and demo, and we recommend it as such on the list > all the time. > > So there isn’t a decision to be made here as far as the status quo -> we > use full dev as both dev and demo. > > > > > On May 2, 2019 at 18:53:37, Michael Miklavcic (michael.miklav...@gmail.com > ) > wrote: > > Whether or not full dev is, first and foremost, "dev" I think your > questions being up a good point. If not full_dev for introducing new users, > then what? If we want to provide a different env for letting people tinker > and try it out than we do for development, that's completely fine. But we > don't have that right now. So we can treat full_dev as multipurpose, or we > can stop directing non-devs to it, or we can add something new. I honestly > don't have any recommendations here. We've talked about docker instances > for replacing in-memory components, but I'm still not sure that solves this > problem, or adds more complexity. Given the current options on the table, > I'm inclined to go with "full_dev" serves both dev and demo purposes. Otto, > what do you think? > > On Thu, May 2, 2019, 4:32 PM Otto Fowler wrote: > > > I’ve commented on the PR, and I won’t repeat it here as well, I will > > however ask again if we know and can list all of the usability issues > that > > surround this problem. IE. All the things that can happen or may happen > > for people who are not Metron developers and committers who are using > > full dev, because we keep recommending it. > > > > > > > > On May 2, 2019 at 17:38:30, Michael Miklavcic ( > michael.miklav...@gmail.com > > ) > > wrote: > > > > PR is up. I added the doc change to the metron-deployment README since > this > > serves as the gateway doc for all the VM instances. All of which would be > > affected by the feature gap. > > > > https://github.com/apache/metron/pull/1398 > > > > On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > Here's the ticket I created to track it, which also references the Jira > > > for the new UI feature. > > > https://issues.apache.org/jira/browse/METRON-2100 > > > > > > On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > >> :-) > > >> > > >> I expect to have #2 out sometime today. > > >> > > >> On Thu, May 2, 2019, 12:11 PM Justin Leet > > wrote: > > >> > > >>> > > > >>> > I personally > > >>> > don't like this feature gap in full dev. It seems Otto agrees, and > > >>> Casey at > > >>> > the very least sees it as enough of an issue to gate us from 0.8. > > >>> > > > >>> > > >>> +1 on all of this. I don't like it either. > > >>> > > >>> > > >>> > Our vote landed 2-2. We are having a discussion about what to do > with > > >>> the > > >>> > release. This is that discussion. > > >>> > > >>> > > >>> I'm going to be honest, my response was a combination of misreading > > what > > >>> you said and thinking you were proposing delaying the release more > > >>> seriously and feeling a bit blindsided by a perceived move from the > > >>> initial > > >>> "take more time than originally anticipated" (which in my head I took > > as > > >>> a > > >>> couple days) to "versus next week, or the week after" (where delaying > > >>> things weeks is something I personally would like not buried so far > > down > > >>> in > > >>> the thread). Totally my bad, sorry about that. > > >>> > > >>> Other than that, it sounds like we're pretty much in agreement. > > >>> > > >>> Here's my current understanding of the state and consensus as of > right > > >>> now > > >>> (which is subject to change as more discussion happens): > > >>> > > >>> - Most of the people in the thread are in favor of #2 for 0.7.1 and > #3 > > >>> for 0.8.0. > > >>> - I don't believe I've seen an explicit response from Otto on what > > >>> he > > >>> thinks about doing this, and from a personal perspective like to > > >>> see what > > >>> his opinion is as the person who originally brought it up. > > >>> - Mike said he's going to kick out a PR that addresses #2 > > >>> - After that undergoes the normal review process and is merged, we > > >>> proceed normally and cut RC2. > > >>> > > >>> > > >>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic < > > >>> michael.miklav...@gmail.com> wrote: > > >>> > > >>> > I think your later point about using a project release version, > from > > >>> the > > >>> > example of using other projects, is a valid one. To exactly that > > >>> point, a > > >>> > community member (Otto) brought up an issue/bug they found through > > >>> testing > > >>> > that they were previously unaware of and did not find documented. > > >>> Which was
[RESULT] [VOTE] Update dev guidelines with format for sharing architecture source files and rendered images
Vote has passed. Voting was: 2 binding +1's (Mike Miklavcic, Otto Fowler) 1 non-binding +1 (Jon Zeolla) I will start making the changes to the dev guidelines today. Vote thread - https://lists.apache.org/thread.html/dccecd41314e76eed7440ab8214ab5ee1ed28724e6f0e545c68ea5e0@%3Cdev.metron.apache.org%3E
Re: [DISCUSS] Parser Aggregation in Management UI
You need to be a committer, that is all I think. I would not use the github UI for it though, I do it through the cli On May 8, 2019 at 09:45:24, Michael Miklavcic (michael.miklav...@gmail.com) wrote: Not that I'm aware of. Nick and Otto, you've created them before, did you need any special perms? On Wed, May 8, 2019 at 3:57 AM Shane Ardell wrote: > This morning, we started to break down our work as Michael suggested in > this thread. However, it looks like I don't have permission to create a new > branch in the GitHub UI or push a new branch to the apache/metron repo. Is > this action restricted to PMC members only? > > Shane > > On Wed, May 8, 2019 at 9:06 AM Tamás Fodor wrote: > > > Here's the process we've gone through in order to implement the feature. > > > > At the beginning we had some bootstrap work like creating a mock API > > (written in NodeJS) because we were a few steps ahead the backend part. > But > > this is in a totally different repository so it doesn't really count. We > > also had to wire NgRX, our chosen 3rd party that supports the flux flow > to > > get a better state management. When we were ready to kick off > implementing > > the business logic in, we splited up the work by subfeatures like drag > and > > dropping table rows. At this point, we created a POC without NgRX just to > > let you have the feeling of how it works in real life. Later on, after > > introducing NgRX, we had to refactor it a little bit obviously to make it > > compatible with NgRX. There were other subfeatures like creating and > > editing groups in a floating pane on the right side of the window. > > When the real backend API was ready we made the necessary changes and > > tested whether it worked how it was expected. There were a few difference > > between how we originally planned the API and the current implementation > so > > we had to adapt it accordingly. While we were implementing the features, > we > > wrote the unit tests simultaneously. The latest task on the feature was > > restricting the user from aggregating parsers together. > > > > As a first iteration, we've decided to put the restriction in because it > > requires a bigger effort on the backend to deal with that. In my opinion, > > we should get rid of the restriction because it's not intuitive and very > > inconvenient. In my opinion, we should let the users to aggregate the > > running parsers together and do the job to handle this edge case on the > > backend accordingly. > > > > What do you think, guys? > > > > Hope this helps. > > > > Tamas > > > > On Tue, May 7, 2019 at 4:34 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > This was my expectation as well. > > > > > > Shane, Tibor, Tamas - how did you go about breaking this down into > chunks > > > and/or microchunks when you collaborated offline? As Nick mentioned, > you > > > obviously split up work and shared it amongst yourselves. Some > > explanation > > > around this process would be helpful for reviewers as well. We might be > > > able to provide better guidance and examples to future contributors as > > > well. > > > > > > I talked a little bit with Shane about this offline last week. It looks > > > like you guys effectively ran a local feature branch. Replicating that > > > process in a feature branch in Apache is probably what you guys should > > > be doing for a change this size. We don't have hard limits on line > change > > > size, but in the past it's been somewhere around 2k-3k lines and above > > > being the tipping point for discussing a feature branch. Strictly > > speaking, > > > line quantity alone is not the only metric, but it's relevant here. If > > you > > > want to make smaller incremental changes locally, there's nothing to > keep > > > you from doing that - I would only advise that you consider squashing > > those > > > commits (just ask if you're unclear about how to handle that) into a > > single > > > larger commit/chunk when you're ready to publish them as a chunk to the > > > public feature branch. So it would look something like this: > > > > > > Commits by person locally > > > Shane: 1,2,3 -> squash as A > > > Tibor: 4,5,6 -> squash as B > > > Tamas: 7,8,9 -> squash as C > > > > > > Commits by person in Apache > > > Shane: A > > > Tibor: B > > > Tamas: C > > > > > > We need to maintain a record of attribution. Your real workflow may not > > be > > > that cleanly delineated, but you can choose how you want to squash in > > those > > > cases. Even in public collaboration, there are plenty of cases where > > folks > > > submit PRs against PRs, abstain from accepting attribution, and it all > > gets > > > squashed down into one person's final PR commit. There are many > options. > > > > > > Hope this helps. > > > > > > Best, > > > Mike > > > > > > On Mon, May 6, 2019 at 8:19 AM Nick Allen wrote: > > > > > > > Have you considered creating a feature branch for the effort? This > > would > > > > allow you to break the effort
Re: [DISCUSS] Parser Aggregation in Management UI
Not that I'm aware of. Nick and Otto, you've created them before, did you need any special perms? On Wed, May 8, 2019 at 3:57 AM Shane Ardell wrote: > This morning, we started to break down our work as Michael suggested in > this thread. However, it looks like I don't have permission to create a new > branch in the GitHub UI or push a new branch to the apache/metron repo. Is > this action restricted to PMC members only? > > Shane > > On Wed, May 8, 2019 at 9:06 AM Tamás Fodor wrote: > > > Here's the process we've gone through in order to implement the feature. > > > > At the beginning we had some bootstrap work like creating a mock API > > (written in NodeJS) because we were a few steps ahead the backend part. > But > > this is in a totally different repository so it doesn't really count. We > > also had to wire NgRX, our chosen 3rd party that supports the flux flow > to > > get a better state management. When we were ready to kick off > implementing > > the business logic in, we splited up the work by subfeatures like drag > and > > dropping table rows. At this point, we created a POC without NgRX just to > > let you have the feeling of how it works in real life. Later on, after > > introducing NgRX, we had to refactor it a little bit obviously to make it > > compatible with NgRX. There were other subfeatures like creating and > > editing groups in a floating pane on the right side of the window. > > When the real backend API was ready we made the necessary changes and > > tested whether it worked how it was expected. There were a few difference > > between how we originally planned the API and the current implementation > so > > we had to adapt it accordingly. While we were implementing the features, > we > > wrote the unit tests simultaneously. The latest task on the feature was > > restricting the user from aggregating parsers together. > > > > As a first iteration, we've decided to put the restriction in because it > > requires a bigger effort on the backend to deal with that. In my opinion, > > we should get rid of the restriction because it's not intuitive and very > > inconvenient. In my opinion, we should let the users to aggregate the > > running parsers together and do the job to handle this edge case on the > > backend accordingly. > > > > What do you think, guys? > > > > Hope this helps. > > > > Tamas > > > > On Tue, May 7, 2019 at 4:34 PM Michael Miklavcic < > > michael.miklav...@gmail.com> wrote: > > > > > This was my expectation as well. > > > > > > Shane, Tibor, Tamas - how did you go about breaking this down into > chunks > > > and/or microchunks when you collaborated offline? As Nick mentioned, > you > > > obviously split up work and shared it amongst yourselves. Some > > explanation > > > around this process would be helpful for reviewers as well. We might be > > > able to provide better guidance and examples to future contributors as > > > well. > > > > > > I talked a little bit with Shane about this offline last week. It looks > > > like you guys effectively ran a local feature branch. Replicating that > > > process in a feature branch in Apache is probably what you guys should > > > be doing for a change this size. We don't have hard limits on line > change > > > size, but in the past it's been somewhere around 2k-3k lines and above > > > being the tipping point for discussing a feature branch. Strictly > > speaking, > > > line quantity alone is not the only metric, but it's relevant here. If > > you > > > want to make smaller incremental changes locally, there's nothing to > keep > > > you from doing that - I would only advise that you consider squashing > > those > > > commits (just ask if you're unclear about how to handle that) into a > > single > > > larger commit/chunk when you're ready to publish them as a chunk to the > > > public feature branch. So it would look something like this: > > > > > > Commits by person locally > > > Shane: 1,2,3 -> squash as A > > > Tibor: 4,5,6 -> squash as B > > > Tamas: 7,8,9 -> squash as C > > > > > > Commits by person in Apache > > > Shane: A > > > Tibor: B > > > Tamas: C > > > > > > We need to maintain a record of attribution. Your real workflow may not > > be > > > that cleanly delineated, but you can choose how you want to squash in > > those > > > cases. Even in public collaboration, there are plenty of cases where > > folks > > > submit PRs against PRs, abstain from accepting attribution, and it all > > gets > > > squashed down into one person's final PR commit. There are many > options. > > > > > > Hope this helps. > > > > > > Best, > > > Mike > > > > > > On Mon, May 6, 2019 at 8:19 AM Nick Allen wrote: > > > > > > > Have you considered creating a feature branch for the effort? This > > would > > > > allow you to break the effort into chunks, where the result of each > PR > > > may > > > > not be a fully working "master-ready" result. > > > > > > > > I am sure you guys tackled the work in chunks when developing it, so > > > >
Re: [DISCUSS] Parser Aggregation in Management UI
This morning, we started to break down our work as Michael suggested in this thread. However, it looks like I don't have permission to create a new branch in the GitHub UI or push a new branch to the apache/metron repo. Is this action restricted to PMC members only? Shane On Wed, May 8, 2019 at 9:06 AM Tamás Fodor wrote: > Here's the process we've gone through in order to implement the feature. > > At the beginning we had some bootstrap work like creating a mock API > (written in NodeJS) because we were a few steps ahead the backend part. But > this is in a totally different repository so it doesn't really count. We > also had to wire NgRX, our chosen 3rd party that supports the flux flow to > get a better state management. When we were ready to kick off implementing > the business logic in, we splited up the work by subfeatures like drag and > dropping table rows. At this point, we created a POC without NgRX just to > let you have the feeling of how it works in real life. Later on, after > introducing NgRX, we had to refactor it a little bit obviously to make it > compatible with NgRX. There were other subfeatures like creating and > editing groups in a floating pane on the right side of the window. > When the real backend API was ready we made the necessary changes and > tested whether it worked how it was expected. There were a few difference > between how we originally planned the API and the current implementation so > we had to adapt it accordingly. While we were implementing the features, we > wrote the unit tests simultaneously. The latest task on the feature was > restricting the user from aggregating parsers together. > > As a first iteration, we've decided to put the restriction in because it > requires a bigger effort on the backend to deal with that. In my opinion, > we should get rid of the restriction because it's not intuitive and very > inconvenient. In my opinion, we should let the users to aggregate the > running parsers together and do the job to handle this edge case on the > backend accordingly. > > What do you think, guys? > > Hope this helps. > > Tamas > > On Tue, May 7, 2019 at 4:34 PM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > This was my expectation as well. > > > > Shane, Tibor, Tamas - how did you go about breaking this down into chunks > > and/or microchunks when you collaborated offline? As Nick mentioned, you > > obviously split up work and shared it amongst yourselves. Some > explanation > > around this process would be helpful for reviewers as well. We might be > > able to provide better guidance and examples to future contributors as > > well. > > > > I talked a little bit with Shane about this offline last week. It looks > > like you guys effectively ran a local feature branch. Replicating that > > process in a feature branch in Apache is probably what you guys should > > be doing for a change this size. We don't have hard limits on line change > > size, but in the past it's been somewhere around 2k-3k lines and above > > being the tipping point for discussing a feature branch. Strictly > speaking, > > line quantity alone is not the only metric, but it's relevant here. If > you > > want to make smaller incremental changes locally, there's nothing to keep > > you from doing that - I would only advise that you consider squashing > those > > commits (just ask if you're unclear about how to handle that) into a > single > > larger commit/chunk when you're ready to publish them as a chunk to the > > public feature branch. So it would look something like this: > > > > Commits by person locally > > Shane: 1,2,3 -> squash as A > > Tibor: 4,5,6 -> squash as B > > Tamas: 7,8,9 -> squash as C > > > > Commits by person in Apache > > Shane: A > > Tibor: B > > Tamas: C > > > > We need to maintain a record of attribution. Your real workflow may not > be > > that cleanly delineated, but you can choose how you want to squash in > those > > cases. Even in public collaboration, there are plenty of cases where > folks > > submit PRs against PRs, abstain from accepting attribution, and it all > gets > > squashed down into one person's final PR commit. There are many options. > > > > Hope this helps. > > > > Best, > > Mike > > > > On Mon, May 6, 2019 at 8:19 AM Nick Allen wrote: > > > > > Have you considered creating a feature branch for the effort? This > would > > > allow you to break the effort into chunks, where the result of each PR > > may > > > not be a fully working "master-ready" result. > > > > > > I am sure you guys tackled the work in chunks when developing it, so > > > consider just replaying those chunks onto the feature branch as > separate > > > PRs. > > > > > > > > > > > > On Mon, May 6, 2019 at 5:24 AM Tibor Meller > > > wrote: > > > > > > > I wondered on the weekend how we could split that PR to smaller > chunks. > > > > That PR is a result of almost 2 months of development and I don't see > > how > > > > to split that to multiple WORKING parts.
