Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated
> I understand that convenience binaries might some issues with uberjars when we go that route for 1.0. But is there any issue with the uberjars as things currently stand? I was under the impression we are OK because we don't distribute them. My impression is that this incorrect and it was the guidance of the mentors that we were indeed in a situation where we bundle dependencies as our build creates uberjars, thus we have to ensure that the jars contain the appropriate notices and license. This was done as part of METRON-531 (see https://github.com/apache/metron/pull/335 for the discussion between Josh Elsner and myself). It is my understanding that independent of whether we release binaries, the fact that our build system builds uber jars, we must ensure that the license and notices are correct within those bundled artifiacts of our build. Mentors, if I'm wrong in this, please let me know, but if I'm right, then it is likely that we need a change in process to keep these license and notices files updated in the individual projects *or* we could choose to stop creating uberjars. Casey On Wed, Sep 12, 2018 at 1:09 PM Michael Miklavcic < michael.miklav...@gmail.com> wrote: > I'm not sure I fully understand what is out of date. I know I have > personally modified our licenses a couple times in the past and used an > automated script that, I believe, Casey Stella had created for doing the > check. I even made some improvements to it a long ways back. It rips > through the maven dependency tree and tells you what isn't in the licenses > file and fails with a non-zero return code. I thought that was part of our > Travis build, or at the very least, the release lifecycle. Is that not the > case, or is there a different context we're talking about here? > > I understand that convenience binaries might some issues with uberjars when > we go that route for 1.0. But is there any issue with the uberjars as > things currently stand? I was under the impression we are OK because we > don't distribute them. It's part of the build, just like tools such as > JUnit, that we don't actually ship. > > Justin - These are the links for guidance that I've found. Is anything else > you've found that we should peruse while figuring this out? > >- https://www.apache.org/dev/licensing-howto.html >- http://www.apache.org/legal/release-policy.html#artifacts > > Mike > > > On Wed, Sep 12, 2018 at 10:29 AM Justin Leet > wrote: > > > Hi all, > > > > As mentioned on the release voting thread, there was a Slack discussion > > around our LICENSE and NOTICE file likely being outdated because they > > haven't been actively kept up to date since graduation. I suggested on > the > > vote thread that we proceed with the current release, but consider it a > > blocker for the next release. > > > > Mentor input on this (and how other projects handle it), would be greatly > > appreciated. > > > > This discussion should result in JIRAs that are brought back to the > thread, > > so we can make sure to track this. > > > > For context, in addition to the standard L management, when we build > > artifacts we shade a lot of jars into a uberjars, thus bundling > > dependencies. However, our current releases are source only, but > > publishing convenience binaries came up in the 1.0 roadmap thread. > > > > I think there are a few things that need to happen to correct our current > > issue and make this easier in the future. > > 1) Get the LICENSE and NOTICE files up to date > > 2) Document the process we went through getting things up to date and > (just > > as importantly) the reasoning behind it. > > 3) Update the PR checklist to include LICENSE and NOTICE files for new > (and > > transitive) dependencies. > > 4) Update or add any processes we need to maintain this properly (e.g. > > release auditing) > > 5) Possibly build tooling for making some of this auditing easier (or use > > existing tool if anyone has suggestions)? > > > > Are there any other steps I'm missing that need to go into JIRAs? > > Any other concerns regarding these files that need to be addressed? > > Any other context I'm missing and that belongs in this discussion? > > >
Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated
So, since NiFi does produce binaries, they require NOTICE and LICENSE updates in two places: - the ‘package’ itself. With nifi usually this is the .nar file ( nars are just jars ). - the nifi-assembly module which builds the .zip binary distribution. It is normal and expected during reviews that committers/reviewers check this. Joe Witt is the final word on it though. Here is an example from a pr I did: https://github.com/apache/nifi/pull/2805#discussion_r196906833 On September 12, 2018 at 14:34:58, Justin Leet (justinjl...@gmail.com) wrote: There is a distinction. The dependencies_with_url.csv does manage to make sure our dependencies (and transitive dependencies) are appropriately accounted for. What we also need to do is make that any changes (if necessary) to the LICENSE and NOTICE files also make it in there. For example, certain attributions may be necessary in the NOTICE file. Similar things can happen with the LICENSE file (e.g. including licenses from dependencies from bundled code). It's possible that we don't have any problems, but I also expect that since we haven't been actively maintaining it that there might be issues. E.g. if the UI has pulled in anything bundled in our source, modifications to the L may need to be made. As far as uberjars go, I believe we're fine. Like you said, we aren't distributing them, so they aren't bundled. Otto, you mentioned on Slack that NiFi requires some checking in having PRs reviewed and in reviewing PRs. Could you share your experience there? On Wed, Sep 12, 2018 at 1:36 PM Otto Fowler wrote: > Are you referring to the dependencies check against the csv? > > > On September 12, 2018 at 13:09:48, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'm not sure I fully understand what is out of date. I know I have > personally modified our licenses a couple times in the past and used an > automated script that, I believe, Casey Stella had created for doing the > check. I even made some improvements to it a long ways back. It rips > through the maven dependency tree and tells you what isn't in the licenses > file and fails with a non-zero return code. I thought that was part of our > Travis build, or at the very least, the release lifecycle. Is that not the > case, or is there a different context we're talking about here? > > I understand that convenience binaries might some issues with uberjars when > we go that route for 1.0. But is there any issue with the uberjars as > things currently stand? I was under the impression we are OK because we > don't distribute them. It's part of the build, just like tools such as > JUnit, that we don't actually ship. > > Justin - These are the links for guidance that I've found. Is anything else > you've found that we should peruse while figuring this out? > > - https://www.apache.org/dev/licensing-howto.html > - http://www.apache.org/legal/release-policy.html#artifacts > > Mike > > > On Wed, Sep 12, 2018 at 10:29 AM Justin Leet > wrote: > > > Hi all, > > > > As mentioned on the release voting thread, there was a Slack discussion > > around our LICENSE and NOTICE file likely being outdated because they > > haven't been actively kept up to date since graduation. I suggested on > the > > vote thread that we proceed with the current release, but consider it a > > blocker for the next release. > > > > Mentor input on this (and how other projects handle it), would be greatly > > appreciated. > > > > This discussion should result in JIRAs that are brought back to the > thread, > > so we can make sure to track this. > > > > For context, in addition to the standard L management, when we build > > artifacts we shade a lot of jars into a uberjars, thus bundling > > dependencies. However, our current releases are source only, but > > publishing convenience binaries came up in the 1.0 roadmap thread. > > > > I think there are a few things that need to happen to correct our current > > issue and make this easier in the future. > > 1) Get the LICENSE and NOTICE files up to date > > 2) Document the process we went through getting things up to date and > (just > > as importantly) the reasoning behind it. > > 3) Update the PR checklist to include LICENSE and NOTICE files for new > (and > > transitive) dependencies. > > 4) Update or add any processes we need to maintain this properly (e.g. > > release auditing) > > 5) Possibly build tooling for making some of this auditing easier (or use > > existing tool if anyone has suggestions)? > > > > Are there any other steps I'm missing that need to go into JIRAs? > > Any other concerns regarding these files that need to be addressed? > > Any other context I'm missing and that belongs in this discussion? > > >
Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated
There is a distinction. The dependencies_with_url.csv does manage to make sure our dependencies (and transitive dependencies) are appropriately accounted for. What we also need to do is make that any changes (if necessary) to the LICENSE and NOTICE files also make it in there. For example, certain attributions may be necessary in the NOTICE file. Similar things can happen with the LICENSE file (e.g. including licenses from dependencies from bundled code). It's possible that we don't have any problems, but I also expect that since we haven't been actively maintaining it that there might be issues. E.g. if the UI has pulled in anything bundled in our source, modifications to the L may need to be made. As far as uberjars go, I believe we're fine. Like you said, we aren't distributing them, so they aren't bundled. Otto, you mentioned on Slack that NiFi requires some checking in having PRs reviewed and in reviewing PRs. Could you share your experience there? On Wed, Sep 12, 2018 at 1:36 PM Otto Fowler wrote: > Are you referring to the dependencies check against the csv? > > > On September 12, 2018 at 13:09:48, Michael Miklavcic ( > michael.miklav...@gmail.com) wrote: > > I'm not sure I fully understand what is out of date. I know I have > personally modified our licenses a couple times in the past and used an > automated script that, I believe, Casey Stella had created for doing the > check. I even made some improvements to it a long ways back. It rips > through the maven dependency tree and tells you what isn't in the licenses > file and fails with a non-zero return code. I thought that was part of our > Travis build, or at the very least, the release lifecycle. Is that not the > case, or is there a different context we're talking about here? > > I understand that convenience binaries might some issues with uberjars when > we go that route for 1.0. But is there any issue with the uberjars as > things currently stand? I was under the impression we are OK because we > don't distribute them. It's part of the build, just like tools such as > JUnit, that we don't actually ship. > > Justin - These are the links for guidance that I've found. Is anything else > you've found that we should peruse while figuring this out? > > - https://www.apache.org/dev/licensing-howto.html > - http://www.apache.org/legal/release-policy.html#artifacts > > Mike > > > On Wed, Sep 12, 2018 at 10:29 AM Justin Leet > wrote: > > > Hi all, > > > > As mentioned on the release voting thread, there was a Slack discussion > > around our LICENSE and NOTICE file likely being outdated because they > > haven't been actively kept up to date since graduation. I suggested on > the > > vote thread that we proceed with the current release, but consider it a > > blocker for the next release. > > > > Mentor input on this (and how other projects handle it), would be greatly > > appreciated. > > > > This discussion should result in JIRAs that are brought back to the > thread, > > so we can make sure to track this. > > > > For context, in addition to the standard L management, when we build > > artifacts we shade a lot of jars into a uberjars, thus bundling > > dependencies. However, our current releases are source only, but > > publishing convenience binaries came up in the 1.0 roadmap thread. > > > > I think there are a few things that need to happen to correct our current > > issue and make this easier in the future. > > 1) Get the LICENSE and NOTICE files up to date > > 2) Document the process we went through getting things up to date and > (just > > as importantly) the reasoning behind it. > > 3) Update the PR checklist to include LICENSE and NOTICE files for new > (and > > transitive) dependencies. > > 4) Update or add any processes we need to maintain this properly (e.g. > > release auditing) > > 5) Possibly build tooling for making some of this auditing easier (or use > > existing tool if anyone has suggestions)? > > > > Are there any other steps I'm missing that need to go into JIRAs? > > Any other concerns regarding these files that need to be addressed? > > Any other context I'm missing and that belongs in this discussion? > > >
Re: [RESULT][VOTE] Metron Release Candidate 0.6.0-RC1
Woop woop! On Wed, Sep 12, 2018 at 12:15 PM Justin Leet wrote: > The vote has passed. Including my +1, the voting was: > 3 binding +1’s > 1 non-binding +1’s > no 0’s > no -1’s. >
Re: [MENTORS][DISCUSS] LICENSE and NOTICE likely outdated
I'm not sure I fully understand what is out of date. I know I have personally modified our licenses a couple times in the past and used an automated script that, I believe, Casey Stella had created for doing the check. I even made some improvements to it a long ways back. It rips through the maven dependency tree and tells you what isn't in the licenses file and fails with a non-zero return code. I thought that was part of our Travis build, or at the very least, the release lifecycle. Is that not the case, or is there a different context we're talking about here? I understand that convenience binaries might some issues with uberjars when we go that route for 1.0. But is there any issue with the uberjars as things currently stand? I was under the impression we are OK because we don't distribute them. It's part of the build, just like tools such as JUnit, that we don't actually ship. Justin - These are the links for guidance that I've found. Is anything else you've found that we should peruse while figuring this out? - https://www.apache.org/dev/licensing-howto.html - http://www.apache.org/legal/release-policy.html#artifacts Mike On Wed, Sep 12, 2018 at 10:29 AM Justin Leet wrote: > Hi all, > > As mentioned on the release voting thread, there was a Slack discussion > around our LICENSE and NOTICE file likely being outdated because they > haven't been actively kept up to date since graduation. I suggested on the > vote thread that we proceed with the current release, but consider it a > blocker for the next release. > > Mentor input on this (and how other projects handle it), would be greatly > appreciated. > > This discussion should result in JIRAs that are brought back to the thread, > so we can make sure to track this. > > For context, in addition to the standard L management, when we build > artifacts we shade a lot of jars into a uberjars, thus bundling > dependencies. However, our current releases are source only, but > publishing convenience binaries came up in the 1.0 roadmap thread. > > I think there are a few things that need to happen to correct our current > issue and make this easier in the future. > 1) Get the LICENSE and NOTICE files up to date > 2) Document the process we went through getting things up to date and (just > as importantly) the reasoning behind it. > 3) Update the PR checklist to include LICENSE and NOTICE files for new (and > transitive) dependencies. > 4) Update or add any processes we need to maintain this properly (e.g. > release auditing) > 5) Possibly build tooling for making some of this auditing easier (or use > existing tool if anyone has suggestions)? > > Are there any other steps I'm missing that need to go into JIRAs? > Any other concerns regarding these files that need to be addressed? > Any other context I'm missing and that belongs in this discussion? >
[MENTORS][DISCUSS] LICENSE and NOTICE likely outdated
Hi all, As mentioned on the release voting thread, there was a Slack discussion around our LICENSE and NOTICE file likely being outdated because they haven't been actively kept up to date since graduation. I suggested on the vote thread that we proceed with the current release, but consider it a blocker for the next release. Mentor input on this (and how other projects handle it), would be greatly appreciated. This discussion should result in JIRAs that are brought back to the thread, so we can make sure to track this. For context, in addition to the standard L management, when we build artifacts we shade a lot of jars into a uberjars, thus bundling dependencies. However, our current releases are source only, but publishing convenience binaries came up in the 1.0 roadmap thread. I think there are a few things that need to happen to correct our current issue and make this easier in the future. 1) Get the LICENSE and NOTICE files up to date 2) Document the process we went through getting things up to date and (just as importantly) the reasoning behind it. 3) Update the PR checklist to include LICENSE and NOTICE files for new (and transitive) dependencies. 4) Update or add any processes we need to maintain this properly (e.g. release auditing) 5) Possibly build tooling for making some of this auditing easier (or use existing tool if anyone has suggestions)? Are there any other steps I'm missing that need to go into JIRAs? Any other concerns regarding these files that need to be addressed? Any other context I'm missing and that belongs in this discussion?
[RESULT][VOTE] Metron Release Candidate 0.6.0-RC1
The vote has passed. Including my +1, the voting was: 3 binding +1’s 1 non-binding +1’s no 0’s no -1’s.
Re: [VOTE] Metron Release Candidate 0.6.0-RC1
Hi all, There was a discussion on Slack that I wanted to bring back to the list regarding our LICENSE and NOTICE files. These files appear to be stale since our graduation to TLP, however they likely should have been updated throughout their lifetime. I'll be starting a discuss thread shortly regarding exactly how we go about getting things into current state, documenting the process, etc. The result of this discuss thread will be a set of JIRAs for this issue. Given that these issues have existed since graduation, I'm inclined to continue with the release as-is and consider it a blocker for the next release. On Tue, Sep 11, 2018 at 6:09 PM James Sirota wrote: > +1 (binding) ran it up in full dev > > 07.09.2018, 06:30, "Justin Leet" : > > This is a call to vote on releasing Apache Metron 0.6.0 and the Bro-Kafka > > plugin 0.2.0 > > > > Full list of changes in this release: > > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES > > > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/CHANGES.bro-plugin > > > > The tags to be voted upon are: > > (apache/metron) apache-metron-0.6.0-rc1 > > (apache/metron-bro-plugin-kafka) apache-metron-bro-plugin-kafka_0.2.0-rc1 > > > > The source archives being voted upon can be found here: > > > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-0.6.0-rc1.tar.gz > > > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/apache-metron-bro-plugin-kafka_0.2.0-rc1.tar.gz > > > > Other release files, signatures and digests can be found here: > > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/ > > > > The release artifacts are signed with the following key: > > https://dist.apache.org/repos/dist/dev/metron/0.6.0-RC1/KEYS > > > > Please vote on releasing this package as Apache Metron 0.6.0-RC1 and the > > Bro-Kafka plugin as 0.2.0-RC1 > > > > When voting, please list the actions taken to verify the release. > > > > Recommended build validation and verification instructions are posted > > here: > > https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds > > > > This vote will be open for until 10pm EDT on Wednesday September 12 2018, > > to account for the weekend. > > > > [ ] +1 Release this package as Apache Metron 0.3.0-RC1 > > > > [ ] 0 No opinion > > > > [ ] -1 Do not release this package because... > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > >
Re: [DISCUSS] Internal Metron fields
Totally agree with replacing dot with something else. We have had so much drama to use either dot or column with ORC either via Hive or Spark. Although we have replaced it with an underscore, it may not be a good idea as it can be confusing with underscores in the internal field names. Cheers, Ali On Wed, Sep 12, 2018 at 8:18 AM James Sirota wrote: > I propose that we just disallow having dots in the field name. Dots seem > to have a special meaning and as we keep adding data stores we may run into > some unintended behavior. We should have logic in our code to check for it > and either auto-correct it (replace with underscores?) or at least throw an > error or a warning. > > Thanks, > James > > 07.09.2018, 16:33, "Ryan Merriman" : > > Internal means it’s not configurable, doesn’t contain our default > separator (dots) and is namespaced with metron. We can definitely improve > on DRY but there’s more to it than that. For example, having 2 different > versions of this field name (ES and Solr) adds a significant amount of > complexity for no real benefit. > > > >> On Sep 7, 2018, at 5:12 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> > >> Can you elaborate on what you mean by "convert to internal?" From your > >> description, it looks like the challenge is from our violations of DRY > when > >> it comes to constants referencing those keys, which would be > eliminated by > >> refactoring. > >> > >>> On Fri, Sep 7, 2018, 3:50 PM Ryan Merriman > wrote: > >>> > >>> I recently worked on a PR that involved changing the default behavior > of > >>> the ElasticsearchWriter to store data using field names with the > default > >>> Metron separator, dots. One of the unfortunate consequences of this is > >>> that although dots are allowed in more recent versions of ES, it > changes > >>> how these fields are stored. Having a dot in a field name causes ES to > >>> treat it as an object field type. We're not quite comfortable with > this > >>> because it could introduce unforeseen side effects that may not be > >>> obvious. Here's the PR: https://github.com/apache/metron/pull/1181 > >>> > >>> As I worked through it I noticed there are a couple fields that > include > >>> separators where it's not actually necessary. They are not nested by > >>> nature and are internal to Metron. The fact that they are internal > means > >>> they show up in constants and are hardcoded in several different > places. > >>> That made the work in the PR above much harder and tedious than it > should > >>> have been. There are 2 in particular that I had to deal with: > source:type > >>> and threat:triage:score in metaalerts. > >>> > >>> Is it worth considering converting these to internal Metron fields so > that > >>> they stay constant and this isn't a problem in the future? I could see > >>> these fields following the same pattern as 'metron_alert'. However > this > >>> would cause pain when upgrading because existing data would need to be > >>> updated with these new fields. > >>> > >>> Just an idea. Curious if other have an opinion on the subject. > > --- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org > > -- A.Nazemian
