[GitHub] metron issue #933: METRON-1452 Rebase Dev Environment on Latest CentOS 6
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/933 @mmiklavc - the special cut has a larger swap volume than the standard image. It also is customized to what Metron required at the time to run. That's pretty much it. If that's still desirable, I'd consider rebuilding the Packer image with a more recent vintage Centos. Otherwise not so much. ---
[GitHub] metron issue #784: METRON-1229: Management UI type is part of the declaratio...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/784 +1 by inspection. ---
[GitHub] metron issue #680: METRON-1078: Metron Indexing fails to stop during "Stop A...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/680 @justinleet (or anyone really), I'd like to push this today, still good? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: METRON-1078: Metron Indexing fails to stop during "Stop A...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/680 @justinleet - this seems better. The change is to not error when the rest service is unavailable but use that as an indication that the topologies are stopped. The rest is code formatting because I've loaded the checkstyle template. :) I'm not 100% happy with how this works, but wanted to get a quick-ish fix out while I got 100% happy. That work is described here: [METRON-1093](https://issues.apache.org/jira/browse/METRON-1093) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/680 Oh yeah, thanks for clarifying, @anandsubbu - I'm not going to change the 'happy path' logic, so when the UI is up, it will work the way it did prior. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: DO NOT MERGE: METRON-1078: Metron Indexing fails to stop ...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/680 I'm starting to think the best approach would to assume the service is stopped if we can't get to the ui (which we kind of do anyway because of how status works). Anybody absolutely hate that idea? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #680: METRON-1078: Metron Indexing fails to stop during "Stop A...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/680 Thanks @justinleet - I'm going to hold off on this, @anandsubbu did some additional testing and it didn't actually solve the problem. :( --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #678: METRON-1076: Management UI status check always reports th...
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/678 +1 by inspection. Good stuff, much improved, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #481: METRON-322 Global Batching and Flushing
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/481 My test environment is being funky, I'm +1 by inspection- thanks for the work, please don't let me hold you up any more. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #481: METRON-322 Global Batching and Flushing
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/481 @mattf-horton - sorry about the hurry up and wait. I'll run this up tomorrow. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #481: METRON-322 Global Batching and Flushing
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/481 Awesome! I can run it up on a live system and see what's what. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #481: METRON-322 Global Batching and Flushing
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/481 @mattf-horton - Can I offer some help getting back to this? Is a pretty important fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #627: METRON-1007 ambari integration for metron-management
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/627 +1 (inspection). Nice one, Otto, thanks. Just bumped into this yesterday. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron issue #614: METRON-992: Create performance tuning guide
Github user dlyle65535 commented on the issue: https://github.com/apache/metron/pull/614 +1. Used this with a largish instance, worked well, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] metron pull request #581: METRON-844: Install Metron Management UI with Amba...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/metron/pull/581#discussion_r118778981 --- Diff: metron-interface/metron-config/scripts/package.json --- @@ -9,7 +9,8 @@ "http-proxy-middleware": "0.17.4", "optimist": "0.6.1", "serve-favicon": "2.4.2", --- End diff -- I'm inclined to agree that we should absolutely track this down, but since you didn't introduce those dependencies in this PR, it doesn't seem right to hold up this PR until we do. I'd recommend opening an additional Jira to address the libraries of concern and treat this as a separate issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #576: METRON-945: Resolve merge conflict in me...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/576 METRON-945: Resolve merge conflict in metron.spec ## Contributor Comments This changeset will allow the rpm build to complete. Currently, master will fail trying to build the rpms. Run full-dev to verify. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [N/A] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? **Note** - you'll need to change network_host to [ _local_, _site_ ] for this to work, I've opened METRON-946 to track this. ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-945 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/576.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #576 commit db009121cd552172e9e6bf7546e1b3df719da3d8 Author: David Lyle <dlyle65...@gmail.com> Date: 2017-05-09T22:13:43Z METRON-945: Resolve merge conflict in metron.spec --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #574: METRON-934: Component and task id are mi...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/574 METRON-934: Component and task id are missing in the indexing topology Hdfs file names. ## Contributor Comments Prior to this changeset, all files written to hdfs will have the format enrichment-null-0-0-timestamp.json. This causes lock errors in environments that use more than 1 task per writer. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [X] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-934 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/574.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #574 commit f24575b4345efde9c07d77b126850743af77ea3d Author: David Lyle <dlyle65...@gmail.com> Date: 2017-05-09T15:35:44Z METRON-934: Component and task id are missing in the indexing topology Hdfs file names. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #563: METRON-840: All "ambari_*" hosts need to...
GitHub user dlyle65535 opened a pull request: https://github.com/apache/incubator-metron/pull/563 METRON-840: All "ambari_*" hosts need to have a /localrepo folder ## Contributor Comments Fixes broken EC2 deployment. With the default config, each host running needs a yum repo created in /localrepo. It can be empty on all but the Metron host. To test, run an EC2 deployment. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed in the root incubating-metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [N/A] Have you written or updated unit tests and or integration tests to verify your changes? - [N/A] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [N/A] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dlyle65535/incubator-metron METRON-840 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-metron/pull/563.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #563 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #561: METRON-913: Create IP Report in Zeppelin
GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/561
METRON-913: Create IP Report in Zeppelin
## Contributor Comments
For a given IP address, produces a report of:
* Most Frequent Connections (Yaf, defaults to 24 hours)
* Recent Connections (Yaf, defaults to 1 hour)
* Top DNS Queries (Bro, defaults to 24 hours)
* All Ports Used (Yaf, defaults to 24 hours)
* HTTP User Agents (Bro, defaults to 24 hours)
## Test Plan
### Setup
The setup is exactly the same as @justinleet described in PR-556, repeated
here:
To test, spin up full-dev. Ensure data flows through for the sensors. For
Yaf, which is not enabled by default, it'll be necessary to start the
sensor-stub
```
service sensor-stubs start yaf
```
It'll also be necessary to add Yaf to the list of sensors run in Ambari. To
do so, stop Metron, edit "Metron Parsers" to include "yaf" (or if the other
sensors already have data output, feel free to just make it "yaf" only).
Let data flow through for a little while (enough so that we can reasonable
make range filters).
Once some data has gone through, we'll need to have an instance of
Zeppelin. Because of the size of the Vagrant instance, we'll want to shut down
unneeded services. Shutdown Metron, Kibana, Storm, Kafka, and HBase.
Install Zeppelin from "Actions - Add Service". It'll prompt you to install
Spark and Hive, do so. Configuration is pretty trivial, all that's necessary
is to set an arbitrary Hive database password. Let this run. The Hive service
check likes to fail on our Vagrant, but it's benign (some impersonation
configuration issue unrelated to actually running our queries). Ignore it and
accept the installation.
From Metron's "Service Actions", run the "Zeppelin Notebook Import", to
load our notebooks into Zeppelin. Use the quick links to navigate to the
Zeppelin UI.
Go into the "Metron IP Report" notebook. Instructions are in the first
paragraph of the readme.
Note: If you've run for less than an hour (which I recommend), the first 2
reports will have the same data.
If you wish to validate the data, use the following queries. The first 4
will validate against Elasticsearch and the final report will validate against
HDFS. This is because we don't store an unanalyzed user_data field.
Run the following from an ssh session to node1 (as root).
### Most Frequent Connections (and Recent Connections)
```
curl -XPOST 'node1:9200/y*/_search?pretty' -H 'Content-Type:
application/json' -d'
{
"size": 0,
"query": {
"filtered": {
"query": {
"match": {
"ip_src_addr": "192.168.138.158"
}
}
}
},
"aggs": {
"ip_dst_addr": {
"terms": {
"field": "ip_dst_addr"
}
}
}
}
'
```
### Top DNS Queries
```
curl -XPOST 'node1:9200/b*/_search?pretty' -H 'Content-Type:
application/json' -d'
{
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"ip_src_addr": "192.168.138.158"
}
},
{
"term": {
"protocol": "dns"
}
}
]
}
},
"aggs": {
"domain_name": {
"terms": {
"field": "query"
}
}
}
}
'
```
### All Ports Used
```
curl -XPOST 'node1:9200/y*/_search?pretty' -H 'Content-Type:
application/json' -d'
{
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"ip_src_addr": "192.168.138.158"
}
}
]
}
},
"aggs": {
"user_agent": {
"terms": { "field" : "ip_dst_port"}
}
}
}
'
```
### HTTP User Agents (verify count of MSIE 8.0 User Agent)
```
hdfs dfs -cat /apps/metron/indexing/indexed/bro/en* | grep http | grep
'MSIE 8.0' | wc -l
```
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please ref
[GitHub] incubator-metron issue #552: METRON-857 FIX: Full_Dev regression: use the pr...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/552 Ditto. +1, thanks Otto! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #459: METRON-726: Clean up mvn site generation
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/459 I think you're suffering from the same intermittent issues we have on master. Concur that you didn't add anything that obviously would create a problem and you may have helped. Maybe measure the pass/fail rate of N runs on this branch and N runs on master and see if they're similar? What was the fail rate for your current testing, btw? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #538: METRON-868 Fix documentation on building RPMs
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/538 +1 - pulled in the dependency changes, deleted my .m2 cache and the build ran as expected. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #544: METRON-866 Quickdev and Fulldev readme m...
Github user dlyle65535 commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/544#discussion_r112950226 --- Diff: metron-deployment/vagrant/full-dev-platform/README.md --- @@ -24,35 +26,38 @@ Any platform that supports these tools is suitable, but the following instructio 1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/). -2. Run the following command in a terminal to install all of the required tools. +1. Run the following command in a terminal to install all of the required tools. - ``` - brew cask install vagrant virtualbox java - brew install maven git - ``` +``` +brew cask install vagrant virtualbox java docker +brew install maven git +``` -3. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip). +1. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip). ### Deploy Metron -1. Install Vagrant Hostmanager. +1. Build Metron - ``` - vagrant plugin install vagrant-hostmanager - ``` +``` +cd incubator-metron +mvn clean install -DskipTests +``` --- End diff -- Metron is built during the provisioning process. There is a defect that makes a single "mvn install" call per release number required. See discussion [here](https://github.com/apache/incubator-metron/pull/538). My hope is the defect will be removed shortly. In the meantime, what would you think of either omitting this or adding some language that makes it clear that the manual install only needs to be called once per release until 538 is resolved (i.e. if we bumped the rev from 0.4.0, install would need to be called again)? Otherwise, following these instructions will add around 13 minutes to each build for no gain. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #543: METRON-857 Ability to completely build p...
Github user dlyle65535 commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/543#discussion_r112808661
--- Diff: metron-deployment/packaging/docker/rpm-docker/pom.xml ---
@@ -227,4 +183,97 @@
+
+
+InDocker
+
+
+
+/.dockerenv
+
+
+
+
+
+org.codehaus.mojo
+exec-maven-plugin
+1.5.0
+
+
+rpm-build
+package
+
+exec
+
+
+/bin/bash
+
+-c
+./build.sh
${project.version}
+
+
+
+
+
+
+
+
+
+NeedsDocker
+
--- End diff --
Won't doing this break the Ansible deployment for quick-dev and full-dev?
Couldn't you just leave the build-rpms profile and add your 'InDocker' profile?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #538: METRON-868 Fix documentation on building RPMs
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/538 Also @mmiklavc, please remember if the intended effect was to make the rpm build happen after everything else completed then install + pom dependencies didn't work. If you take a close look during the multi-threaded build, you'll see docker-build output interleaved with the output from the other module builds. e.g. (from a mvn clean install -Pbuild-rpms run) ``` INFO] [INFO] --- maven-shade-plugin:2.4.3:shade (default) @ elasticsearch-shaded --- [INFO] [INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-wait-for-it-to-hbase) @ metron-docker --- [INFO] Executing tasks [echo] Displaying value of property [echo] ../../../.. [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-rpm-sources) @ metron-rpm --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-wait-for-it-to-kafkazk) @ metron-docker --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] [INFO] --- maven-resources-plugin:3.0.1:copy-resources (copy-wait-for-it-to-elasticsearch) @ metron-docker --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] ``` In this example, you'll see that metron-rpm and metron-docker are happening at the same time at the beginning of the build even though they're later in the build order. ``` [INFO] metron-writer [INFO] metron-storm-kafka [INFO] metron-hbase [INFO] metron-profiler-common [INFO] metron-profiler-client [INFO] metron-profiler [INFO] metron-enrichment [INFO] metron-indexing [INFO] metron-solr [INFO] metron-pcap [INFO] metron-parsers [INFO] metron-pcap-backend [INFO] metron-data-management [INFO] metron-api [INFO] metron-management [INFO] elasticsearch-shaded [INFO] metron-elasticsearch [INFO] metron-deployment [INFO] metron-rpm [INFO] metron-docker ``` Here's the output from the build that shows the actual build executed order: ``` [INFO] Building metron-analytics 0.4.0 [INFO] Building metron-docker 0.4.0 [INFO] Building metron-interface 0.4.0 [INFO] Building metron-platform 0.4.0 [INFO] Building metron-maas-common 0.4.0 [INFO] Building metron-test-utilities 0.4.0 [INFO] Building metron-deployment 0.4.0 [INFO] Building metron-config 0.4.0 [INFO] Building metron-rpm 0.4.0 [INFO] Building metron-integration-test 0.4.0 [INFO] Building metron-maas-service 0.4.0 [INFO] Building metron-common 0.4.0 [INFO] Building metron-rest-client 0.4.0 [INFO] Building metron-statistics 0.4.0 [INFO] Building metron-writer 0.4.0 [INFO] Building metron-hbase 0.4.0 [INFO] Building metron-storm-kafka 0.4.0 [INFO] Building metron-profiler-common 0.4.0 [INFO] Building metron-pcap 0.4.0 [INFO] Building metron-profiler-client 0.4.0 [INFO] Building metron-pcap-backend 0.4.0 [INFO] Building metron-api 0.4.0 [INFO] Building metron-profiler 0.4.0 [INFO] Building metron-enrichment 0.4.0 [INFO] Building metron-indexing 0.4.0 [INFO] Building metron-parsers 0.4.0 [INFO] Building metron-data-management 0.4.0 [INFO] Building metron-elasticsearch 0.4.0 [INFO] Building metron-solr 0.4.0 [INFO] Building metron-management 0.4.0 [INFO] Building metron-rest 0.4.0 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #538: METRON-868 Fix documentation on building RPMs
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/538 In fact, having given it additional thought, the dependency mechanism is a way of saying "hey Maven, make sure you pull those artifacts into our .m2 cache" which isn't really the desired result here. We need to make sure the reactor orderer understands that we cannot build the RPMs until we've built the other tar.gzs. The reason it started becoming a problem was that we didn't have the 0.4.0 artifacts in our m2 cache so Maven complained. Running an install "fixed" the issue but, we don't need them there to build the rpms, so not really a fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #540: METRON-869 Include build instructions for full-...
Github user dlyle65535 commented on the issue: https://github.com/apache/incubator-metron/pull/540 You shouldn't have to build Metron prior to running 'vagrant up', the Ansible provisioner does that. Following these instructions, you'll actually build Metron twice for each build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
