[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 To confirm @mmiklavc about maven, I did hit the exact same issue using 3.5.0, although I didn't validate with 3.3.9. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/624 I added a fix to actually clear out the correct directory of Maven artifacts before caching. In a separate, experimental branch, there's an attempt to cache the artifacts resulting from npm. See: https://github.com/justinleet/metron/tree/caching and https://travis-ci.org/justinleet/metron. This required a run without the integration tests on in order to make it to the populating the cache successfully, then reenabling them next commit. At this point we do have intermittent successful builds on my Travis, although I'm doubt it's consistent. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [VOTE] Apache Metron 0.4.0 release
➢ Should this bother us? Yes and no. The important message is “Good signature from ”. To validate my key, compare it’s fingerprint with this authoritative database: https://people.apache.org/keys/committer/ . Compare the full 40-character key fingerprint, not just the last 8 characters. WARNING: In the good old days we used an 8-character (4-byte) fingerprint to id the key, and figured that was sufficient. Well, the MIT Public Key Server (https://pgp.mit.edu/ ) contains evidence that it isn’t. There are fake keys for many Apache people, including myself (!), in there alongside our real ones, with the SAME 8-character fingerprint, signed by fakes of the same people who signed my real one, also with matching 8-char fingerprints. (The fakes were revoked en masse last year when found.) What you have to do is use AT LEAST 16-character and preferably the full 40-character fingerprints, which are exponentially more difficult to fake. The 16-char fingerprint of fake me in the MIT key server (6128A936 ECB31663) is different than my real 16-char fingerprint (4169AA27 ECB31663). The warning about “This key is not certified with a trusted signature” has to do with the state of your personal “web of trust” on your local server, not the state of my signature key. A full technical discussion of the problem from Apache is here: https://www.apache.org/dev/release-signing.html (the info, including definition of “web of trust” and how you get one) and here: https://httpd.apache.org/dev/verification.html (the nasty potential problem with key verification with short fingerprints). The latter doc gives the link to an objective place you can confirm Apache committer keys: https://people.apache.org/keys/committer/ . Apache INFRA has secured this page as best they can. The best thing is still to establish your own web of trust, which mostly can’t be hacked. TL;DR, for a shorter explanation try here: https://security.stackexchange.com/questions/147447/gpg-why-is-my-trusted-key-not-certified-with-a-trusted-signature . Basically, a key is only trusted if your and my webs of trust have a trusted overlap. In particular, my key is signed by Owen O’Malley and Mahadev Konar. Your personal gpg key (you have one, right? :-) would have to have signed / been signed by myself and/or one of Owen or Mahadev, in order for my key to be directly “trusted” on your system. This can be expanded through transitive trust, hence the “web” of trust – but you have to have loaded all the trusted keys in between into your keychain or gpg db on your local server. There’s also a possible message “gpg: no ultimately trusted keys found”, which is a matter of whether you’ve expressed that level of trust by manually adding it to your gpg db – which there is no requirement for you to do, although some key generators automatically express “ultimate trust” in your own key when you add it to your keyring. Cheers, --Matt On 6/27/17, 3:04 PM, "Otto Fowler" wrote: Matt, Should this bother us? gpg: Signature made Tue Jun 27 13:50:58 2017 EDT using RSA key ID ECB31663 gpg: Good signature from "Matthew Foley (CODE SIGNING KEY) " [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 On June 27, 2017 at 14:36:07, Matt Foley (ma...@apache.org) wrote: This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”. (Note: this is rc4 because the release candidate needed to be modified with another commit after the rc3 tag was pushed to public.) Full list of changes in this release: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES The tag/commit to be voted upon is: d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4) The source archive being voted upon can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/apache-metron-0.4.0-rc4.tar.gz and in github at: https://github.com/apache/metron/tree/Metron_0.4.0 Other release files, signatures and digests can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS The release artifacts are signed with the following key: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA] Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 uid = Matthew Foley (CODE SIGNING KEY) Please vote on releasing this package as Apache Metron 0.4.0. When voting, please list the actions taken to verify the release. Recommended build validation and verification instructions are posted here: https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds This vote will be open
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/530 Spoke with @ottobackwards in IRC channel - I'm running Maven 3.5.0. I swapped back to 3.3.9 to check this and it works fine. I'll see if I can figure out what's wrong in 3.5 if it's something simple enough we can fix here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Start of the deployment readme, parser deployment to follow. > NOTE: METRON-942 has some changes in this area, since there where fixes when actually writing the 3rd party extension installer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [VOTE] Apache Metron 0.4.0 release
+1 * verified keys * ran : mvn -q -T 2C -DskipTests install && \ ] 6:04 PM mvn -q -T 2C org.jacoco:jacoco-maven-plugin:prepare-agent surefire:test@unit-tests && mvn -q \ org.jacoco:jacoco-maven-plugin:prepare-agent surefire:test@integration-tests && mvn -q \ org.jacoco:jacoco-maven-plugin:prepare-agent test --projects metron-interface/metron-config && \ build_utils/verify_licenses.sh * Full Dev Vagrant * verified: ambari storm-ui metron-ui metron-rest metron-management-ui On June 27, 2017 at 14:36:07, Matt Foley (ma...@apache.org) wrote: This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”. (Note: this is rc4 because the release candidate needed to be modified with another commit after the rc3 tag was pushed to public.) Full list of changes in this release: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES The tag/commit to be voted upon is: d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4) The source archive being voted upon can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/apache-metron-0.4.0-rc4.tar.gz and in github at: https://github.com/apache/metron/tree/Metron_0.4.0 Other release files, signatures and digests can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS The release artifacts are signed with the following key: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA] Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 uid = Matthew Foley (CODE SIGNING KEY) Please vote on releasing this package as Apache Metron 0.4.0. When voting, please list the actions taken to verify the release. Recommended build validation and verification instructions are posted here: https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds This vote will be open for at least 72 hours. Please vote one of the following responses: +1 Release this package as Apache Metron 0.4.0-RC4 0 No opinion -1 Do not release this package because... Thank you, --Matt (your friendly release manager)
Re: [VOTE] Apache Metron 0.4.0 release
Matt, Should this bother us? gpg: Signature made Tue Jun 27 13:50:58 2017 EDT using RSA key ID ECB31663 gpg: Good signature from "Matthew Foley (CODE SIGNING KEY) " [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 On June 27, 2017 at 14:36:07, Matt Foley (ma...@apache.org) wrote: This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”. (Note: this is rc4 because the release candidate needed to be modified with another commit after the rc3 tag was pushed to public.) Full list of changes in this release: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES The tag/commit to be voted upon is: d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4) The source archive being voted upon can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/apache-metron-0.4.0-rc4.tar.gz and in github at: https://github.com/apache/metron/tree/Metron_0.4.0 Other release files, signatures and digests can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS The release artifacts are signed with the following key: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA] Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 uid = Matthew Foley (CODE SIGNING KEY) Please vote on releasing this package as Apache Metron 0.4.0. When voting, please list the actions taken to verify the release. Recommended build validation and verification instructions are posted here: https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds This vote will be open for at least 72 hours. Please vote one of the following responses: +1 Release this package as Apache Metron 0.4.0-RC4 0 No opinion -1 Do not release this package because... Thank you, --Matt (your friendly release manager)
[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...
Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #586: METRON-508 Expand Elasticsearch templates to suppo...
GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin # Look at the storm logs (The "failed to parse" errors for ip_src_addr and ip_dst_addr are expected, and should be addressed as a part of METRON-939) tail -
Re: [VOTE] Apache Metron 0.4.0 release
Sorry, small typo in “Other release files..found here” link, corrected below. On 6/27/17, 11:36 AM, "Matt Foley" wrote: This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”. (Note: this is rc4 because the release candidate needed to be modified with another commit after the rc3 tag was pushed to public.) Full list of changes in this release: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES The tag/commit to be voted upon is: d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4) The source archive being voted upon can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/apache-metron-0.4.0-rc4.tar.gz and in github at: https://github.com/apache/metron/tree/Metron_0.4.0 Other release files, signatures and digests can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/ The release artifacts are signed with the following key: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA] Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 uid = Matthew Foley (CODE SIGNING KEY) Please vote on releasing this package as Apache Metron 0.4.0. When voting, please list the actions taken to verify the release. Recommended build validation and verification instructions are posted here: https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds This vote will be open for at least 72 hours. Please vote one of the following responses: +1 Release this package as Apache Metron 0.4.0-RC4 0 No opinion -1 Do not release this package because... Thank you, --Matt (your friendly release manager)
[VOTE] Apache Metron 0.4.0 release
This is a call to vote on releasing this rc4 as “Apache Metron 0.4.0”. (Note: this is rc4 because the release candidate needed to be modified with another commit after the rc3 tag was pushed to public.) Full list of changes in this release: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/RELEASE_NOTES The tag/commit to be voted upon is: d52f574f8294e453ecad3871526858a0c3c2033d (tag apache-metron-0.4.0-rc4) The source archive being voted upon can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/apache-metron-0.4.0-rc4.tar.gz and in github at: https://github.com/apache/metron/tree/Metron_0.4.0 Other release files, signatures and digests can be found here: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS The release artifacts are signed with the following key: https://dist.apache.org/repos/dist/dev/metron/0.4.0-RC4/KEYS pub rsa4096/4169AA27ECB31663 2011-07-31 [SCEA] Key fingerprint = 7854 36A7 8258 6B71 829C 67A0 4169 AA27 ECB3 1663 uid = Matthew Foley (CODE SIGNING KEY) Please vote on releasing this package as Apache Metron 0.4.0. When voting, please list the actions taken to verify the release. Recommended build validation and verification instructions are posted here: https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds This vote will be open for at least 72 hours. Please vote one of the following responses: +1 Release this package as Apache Metron 0.4.0-RC4 0 No opinion -1 Do not release this package because... Thank you, --Matt (your friendly release manager)
[GitHub] metron pull request #625: METRON-1006 Remove Incubator DISCLAIMER file and f...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/625 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #625: METRON-1006 Remove Incubator DISCLAIMER file and fix Rele...
Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/625 This has been +1'ed in email by the following reviewers: @nickwallen : +1 Yes, definitely cruft. Good find. @cestella : +1 to removing it. Other top level projects do not have a disclaimer (see, for example, hbase: http://www-eu.apache.org/dist/hbase/stable) @ottobackwards : +1 Thanks, all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #625: METRON-1006 Remove Incubator DISCLAIMER file and f...
GitHub user mattf-horton opened a pull request: https://github.com/apache/metron/pull/625 METRON-1006 Remove Incubator DISCLAIMER file and fix Release Process doc ## Contributor Comments The Release Process document at https://cwiki.apache.org/confluence/display/METRON/Release+Process currently states that a file named DISCLAIMER is a required artifact in the release package. The contents of this file are: ``` Apache Metron is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. ``` This clearly seems to be left over from our incubation days. I propose to entirely remove the DISCLAIMER file, from the source tree and from the Release Process document. The usual disclaimers remain in the LICENSE file, of course. ## Pull Request Checklist ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: NA ### For documentation related changes: NA You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf-horton/metron Metron_0.4.0 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/625.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #625 commit a744604fa925aa329d6242286c2737f0942ab2c8 Author: mattf-horton Date: 2017-06-27T17:13:10Z METRON-1006 Remove Incubator DISCLAIMER file and fix Release Process doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Metron Release rules
Right, I looked and found nothing that seemed comparable for TLPs. Thanks, I’ll proceed. --Matt On 6/27/17, 2:31 AM, "Casey Stella" wrote: +1 to removing it. Other top level projects do not have a disclaimer (see, for example, hbase: http://www-eu.apache.org/dist/hbase/stable) On Tue, Jun 27, 2017 at 4:40 AM, Otto Fowler wrote: > Is there an equivalent disclaimer for full products? > > On June 26, 2017 at 21:53:34, Nick Allen (n...@nickallen.org) wrote: > > > +1 Yes, definitely cruft. Good find. > > > > On Mon, Jun 26, 2017 at 9:50 PM, Matt Foley wrote: > > > > The Release Process document at https://cwiki.apache.org/ > > confluence/display/METRON/Release+Process > > currently states that a file named DISCLAIMER is a required artifact in > > the release package. The contents of this file are: > > === > > Apache Metron is an effort undergoing incubation at The Apache Software > > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > > required > > of all newly accepted projects until a further review indicates that the > > infrastructure, communications, and decision making process have > > stabilized in > > a manner consistent with other successful ASF projects. While incubation > > status > > is not necessarily a reflection of the completeness or stability of the > > code, > > it does indicate that the project has yet to be fully endorsed by the > ASF. > > === > > This clearly seems to be left over from our incubation days. I propose to > > entirely remove the DISCLAIMER file, from the source tree and from the > > Release Process document. > > > > The usual disclaimers remain in the LICENSE file, of course. > > > > Any objections? > > Thanks, > > --Matt > > > > > > > > > > > > > > >
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 OK, IWOMM. I'm on irc if you want to jump on --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/530 Looking into it now, but building locally I see this: ``` $ mvn clean install -DskipTests -T 2C ... [INFO] elasticsearch-shaded ... SKIPPED [INFO] metron-elasticsearch ... SKIPPED [INFO] metron-maven-parser-extension-archetype FAILURE [ 0.995 s] [INFO] metron-maven-archetypes SKIPPED [INFO] metron-deployment .. SKIPPED [INFO] Metron Ambari Management Pack .. SKIPPED [INFO] metron-docker .. SKIPPED [INFO] metron-interface ... SKIPPED [INFO] metron-config .. SKIPPED [INFO] metron-rest-client . SKIPPED [INFO] metron-rest SKIPPED [INFO] site-book .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 2.138 s (Wall Clock) [INFO] Finished at: 2017-06-27T09:51:59-06:00 [INFO] Final Memory: 38M/418M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.0.0:integration-test (default-integration-test) on project metron-maven-parser-extension-archetype: [ERROR] Archetype IT 'basic' failed: org.codehaus.plexus.util.xml.pull.XmlPullParserException: end tag not allowed in epilog but got / (position: END_TAG seen ...\n\n\n [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :metron-maven-parser-extension-archetype ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/624 Most recent commit, as @ottobackwards points out, merits some explanation. Specifically it turns off the jacoco:prepare-agent from the Travis build and allows us to just run the tests directly without it. We don't actually produce the reports here, so it's pretty extraneous. Locally this resulted in: ``` mvn -q -T 2C surefire:test@unit-tests 309.46s user 20.20s system 169% cpu 3:14.92 total mvn -q -T 2C jacoco:prepare-agent surefire:test@unit-tests 555.20s user 25.93s system 254% cpu 3:47.90 total ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...
Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/624#discussion_r124317746 --- Diff: .travis.yml --- @@ -17,7 +17,7 @@ before_install: - export PATH=$M2_HOME/bin:$PATH script: - | --- End diff -- Sorry, my bad. Usually I tend to consider the commits less important because it's usually a full feature, and it's just minor changes / fixes afterwards. I'll try to make sure the messages are easier to follow, since this is pretty ongoing until it's consistent. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...
Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/624#discussion_r124314669 --- Diff: .travis.yml --- @@ -17,7 +17,7 @@ before_install: - export PATH=$M2_HOME/bin:$PATH script: - | --- End diff -- because it makes it slow right? Can we document with the commits, as you go, the rationale behind the changes, so we can look back and understand a little bit? "why did we get rid of FOO?" Let me check the commit log > " Remove foo. It is seen to cause an increase of X in Y and do z. it is also pretty snarky and fresh" "Oh, that makes sense" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceed...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/624#discussion_r124314003
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
---
@@ -90,23 +90,6 @@ public boolean accept(File dir, String name) {
}
--- End diff --
I think until this is completely deprecated, we should keep the test, but
disable it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I have added more documentation and I'm working on documenting where things are deployed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #619: METRON-939 Elasticsearch ES5 with Xshield client support
Github user wardbekker commented on the issue: https://github.com/apache/metron/pull/619 hey @cestella, @simonellistonball, see updated contributor notes. It's not ready for a official pull request, but this gives a good idea on the impact on the code for a working ES5.x implementation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...
Github user justinleet commented on the issue: https://github.com/apache/metron/pull/624 As a note, what I have is currently the first steps towards reusing infra. It's not perfect, and it's not reused across classes. There was an attempt to use the build matrix to split fast and slow tests, but it resulted in inconsistent failures. Seems like Maven gets tangled up between the builds. Could merit further investigation. it'll increase processing time (because both unit and integration tests have to actually build), but should avoid having either portion of the build timeout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/624 I submitted PRs against this branch to incorporate the suggested changes above for: * Selective shading for non-leaf projects to cut the build times dramatically. * `TaxiiIntegrationTest` * `PcapIntegrationTest` I submit them without credit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #530: METRON-777 Metron Extension System and Parser Extensions
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Thank you @mattf-horton, I was able to fix the links. So, @mmiklavc et. al., can we frame our documentation discussion around filling out and improving these documents? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: Metron Release rules
+1 On June 27, 2017 at 05:31:22, Casey Stella (ceste...@gmail.com) wrote: +1 to removing it. Other top level projects do not have a disclaimer (see, for example, hbase: http://www-eu.apache.org/dist/hbase/stable) On Tue, Jun 27, 2017 at 4:40 AM, Otto Fowler wrote: > Is there an equivalent disclaimer for full products? > > On June 26, 2017 at 21:53:34, Nick Allen (n...@nickallen.org) wrote: > > > +1 Yes, definitely cruft. Good find. > > > > On Mon, Jun 26, 2017 at 9:50 PM, Matt Foley wrote: > > > > The Release Process document at https://cwiki.apache.org/ > > confluence/display/METRON/Release+Process > > currently states that a file named DISCLAIMER is a required artifact in > > the release package. The contents of this file are: > > === > > Apache Metron is an effort undergoing incubation at The Apache Software > > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > > required > > of all newly accepted projects until a further review indicates that the > > infrastructure, communications, and decision making process have > > stabilized in > > a manner consistent with other successful ASF projects. While incubation > > status > > is not necessarily a reflection of the completeness or stability of the > > code, > > it does indicate that the project has yet to be fully endorsed by the > ASF. > > === > > This clearly seems to be left over from our incubation days. I propose to > > entirely remove the DISCLAIMER file, from the source tree and from the > > Release Process document. > > > > The usual disclaimers remain in the LICENSE file, of course. > > > > Any objections? > > Thanks, > > --Matt > > > > > > > > > > > > > > >
[GitHub] metron issue #624: [DO NOT MERGE] METRON-1004: Travis CI - Job Exceeded Maxi...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/624 This does look good. A couple of observations in no particular order of importance; just wanted to get this out there for discussion. # Considering the overhead I want to consider the overhead not in our tests for a moment. In the last run, I count the following timings: * build - 5:41 * unit tests - 2:59 * integration tests - 14:44 * metron-config - 2:17 * verify licenses - 0:16 That's 25:57 out of a total run from Travis of 31:53, which is 5:56 overhead. We should factor that in. # Where to Focus ## Build Time The natural conclusion is to focus on the long pole, those integration tests, but we may be served to also consider the build time. Our build takes a long time and we depend upon parallelization to make the build return in a sensible time (the user time for the build is 26 minutes!). Furthermore, our build is extremely IO heavy due to the shading that we (necessarily) do. While we are on a shared system with the rest of the apache projects, I think reducing the IO burden of our build. While I think that shading is important, we have a very ham-fisted way of doing it. We shade for two reasons: * Relocation of dependencies that conflict * Creating uber jars for Storm One issue is that if we consider the tree of projects induced by their dependent nature, is that we shade non-leaf projects for purpose of relocation. I propose we stop doing that. Let's take, for instance, `metron-common`. We shade that project to relocate guava and beanutils. The consequences of relocating 2 packages is 47M of dependencies. Those 47M of dependencies also gets bundled again into all of the leaf projects (e.g. `metron-parsers`, etc.), thus shading twice. I propose fixing this one of two ways: * aggressively exclude ALL dependencies other than `org.apache.metron` and the relocated dependencies in any project that needs shading purely for relocation * Extract the shaded/relocated dependencies across the project into a separate project and make all of our non-leaf dependencies non-shaded I think the first may be the easiest to achieve and most surgical. Ultimately, it may even be advantageous to have a single jar created as the deployable output of our process (or maybe a small handful representing the independent subcomponents: `metron`, `MaaS` and `stellar`). ## Integration Tests Obviously the integration tests are the long pole in the tent. A couple of thoughts on these: ### `TaxiiIntegrationTest` My impression was that it was slow because parsing taxii via the mitre library was downright arduous. It costs us ~2:30 as of the working build above. We are passing a relatively large blob of taxii in and should consider trimming the taxii example data down to something more manageable and see if that will drop the timing down. ### `PcapIntegrationTest` We currently test two modes for the PcapIntegrationTest, pulling the timestamp from the key and pulling the timestamp from the message itself. We know that in production, we only want to support pulling the timestamp from the key. We might cut this test time in half by only testing the supported approach (it's 81 seconds as of last count). ### `Parser Integration Tests` We might want to reconsider what we integration test here. We currently have an integration test for every parser and we may get the same coverage by mocking out the `ParserWriterBolt` and constructing a shim to pass data in, execute against the real parser bolt, capture data written and evaluate the output. This would drop the overhead for each parser test dramatically (no storm or kafka) and would keep the semantics of the tests. Admittedly this may not be a focus in terms of bang-for-buck because total parser cost is around 86 seconds. # Reuse Integration Test Infrastructure This seems to be the persistent conversation whenever our tests start to push us over the edge. We incur quite a bit of overhead because we spin up and down integration test infrastructure in our `InMemoryComponent`s. We could consider correcting this in a couple of ways: * Reusing the infrastructure * Either use the in memory components or spin up light weight versions of the infrastructure and then run the tests against that (i.e. docker or separate-process versions of the in-memory components). * We'd need to refactor each integration test to clean up after itself so other tests are not splashed * Parallelizing the Integration Tests * Have the `InMemoryComponent`s be able to run in parallel * This would require refactoring the components to seek for open ports and use them. These are just my thoughts that I wanted to get o
Re: Metron Release rules
+1 to removing it. Other top level projects do not have a disclaimer (see, for example, hbase: http://www-eu.apache.org/dist/hbase/stable) On Tue, Jun 27, 2017 at 4:40 AM, Otto Fowler wrote: > Is there an equivalent disclaimer for full products? > > On June 26, 2017 at 21:53:34, Nick Allen (n...@nickallen.org) wrote: > > > +1 Yes, definitely cruft. Good find. > > > > On Mon, Jun 26, 2017 at 9:50 PM, Matt Foley wrote: > > > > The Release Process document at https://cwiki.apache.org/ > > confluence/display/METRON/Release+Process > > currently states that a file named DISCLAIMER is a required artifact in > > the release package. The contents of this file are: > > === > > Apache Metron is an effort undergoing incubation at The Apache Software > > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > > required > > of all newly accepted projects until a further review indicates that the > > infrastructure, communications, and decision making process have > > stabilized in > > a manner consistent with other successful ASF projects. While incubation > > status > > is not necessarily a reflection of the completeness or stability of the > > code, > > it does indicate that the project has yet to be fully endorsed by the > ASF. > > === > > This clearly seems to be left over from our incubation days. I propose to > > entirely remove the DISCLAIMER file, from the source tree and from the > > Release Process document. > > > > The usual disclaimers remain in the LICENSE file, of course. > > > > Any objections? > > Thanks, > > --Matt > > > > > > > > > > > > > > >
