Hi, I am going through the documentation but I feel that I need to pick up knowledge on some of the components used. The sensors do not seem to be started as well. When I vagrant up the full-dev environment the below seems to indicate to me the sensors are skipped: Running with ansible-skip-tags: ["sensors", "quick_dev"]
As I would like to assist with the parser development I thought it might make sense to beef up on my Nifi, Kafka and Storm knowledge. Any other things that I should look to help me get to grips quickly with the platform? Is Nifi installed as well with the full-dev install? Regards, Mark de Rijk From: Mark de Rijk <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Tuesday, 25April, 2017 at 07:15 To: Otto Fowler <[email protected]>, Nick Allen <[email protected]>, "[email protected]" <[email protected]> Subject: Re: Failure installing Full-Dev Vagrant VM Hi, To vouch for Otto. 1. Yes do check if docker is installed and running before even attempting to install. 2. Script from Otto will clean up after this Pebkac action by me. Now I have to figure out the sensors so I can setup and monitor the Bro, YAF and Snort sensors. Once I got all those working I want to work on seeing how I can bring data in from devices that perform syslogging so I can start thinking about actual parser development. Fortunately, due to different clients I can capture different datasets, anonymize and work from there. Any book recommendations on what is used for the data ingestion layer are most welcome as well. As this is my first open source project I am looking to contribute to I might be a little slow compared to you all but I am determined to be of value to the community. With my background in SecOps I thought Metron is a natural fit for a contribution target. Regards, Mark de Rijk From: Otto Fowler <[email protected]> Date: Monday, 24April, 2017 at 14:53 To: Nick Allen <[email protected]>, "[email protected]" <[email protected]> Cc: Mark de Rijk <[email protected]> Subject: Re: FW: Failure installing Full-Dev Vagrant VM Nick, I think there were two problems: 1. Docker not installed and running 2. Subsequent to that, mvn clean from the top level does not clean the rpm-docker project, because it is only included if the -P build-rpms profile is active, thus rat-check pukes on the rpm generated objects. My script cleans both. On April 24, 2017 at 09:46:48, Nick Allen ([email protected]<mailto:[email protected]>) wrote: Mark - What lessons did we learn here? What was the core problem? Do we need to update anything to help others avoid the issues that you ran into? On Sun, Apr 23, 2017 at 4:38 PM, Otto Fowler <[email protected]<mailto:[email protected]>> wrote: Ok the support scripts should not be in the incubator-metron folder move them out and run the scripts again. On April 23, 2017 at 16:05:04, Mark de Rijk ([email protected]<mailto:[email protected]>) wrote: Hi, I am still struggling with the full-dev install. I have run the support scripts at : https://github.com/ottobackwards/Metron-and-Nifi-Scripts, > export METRON_HOME=/SOMEWHERECOOL/src/apache/forks/incubator-metron > support/real-clean-metron | tee what_happened.log && > support/install-build-all-metron | tee what_happened.log I have attached the log and the rat file which was created this time. Regards, Mark de Rijk On April 23, 2017 at 09:16:40, Otto Fowler ([email protected]<mailto:[email protected]>) wrote: #apache-metron on freenode. I’ll email you something to try and you can get back to the list. On April 23, 2017 at 09:14:44, Mark de Rijk ([email protected]<mailto:[email protected]>) wrote: Hi, I am out for the afternoon but I will be back online around 7:30 pm or so GMT. Do you have the IRC details so I can join the channel later? As said I want to get the environment up and running as soon as possible so I can start contributing to the project. Regards, Mark de Rijk From: Otto Fowler <[email protected]<mailto:[email protected]>> Date: Sunday, 23April, 2017 at 14:09 To: Mark de Rijk <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Failure installing Full-Dev Vagrant VM Good morning Mark, I’m going to read through your mails again. Maybe you can jump on irc? On April 23, 2017 at 09:02:03, Mark de Rijk ([email protected]<mailto:[email protected]>) wrote: Hi, As the target directory and file is not there what can I do to make the installation progress? Is there are separation installation switch I have to supply? I read something on the Dev list about switching out Maven but how would I go about doing that? Regards, Mark de Rijk From: Nick Allen <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, 23April, 2017 at 13:56 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Failure installing Full-Dev Vagrant VM The 'target' directory must have been removed at some point. Continue your progress. Any problems? On Sat, Apr 22, 2017 at 7:56 PM, Mark de Rijk <[email protected]<mailto:[email protected]>> wrote: Hi, I was already searching for the target directory but it doesn’t seem to exist: /Users/mark/Applications/Vagrant/incubator-metron-master/metron-deployment metron-deployment mark$ ls README.md ansible.cfg inventory playbooks roles vagrant amazon-ec2 extra_modules packaging pom.xml scripts Regards, Mark de Rijk From: Ryan Merriman <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, 23April, 2017 at 00:43 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Failure installing Full-Dev Vagrant VM Mark, We have a maven plugin that checks all the files in incubator-metron for a license header. You need to remove the offending file (will be listed in /Users/mark/Applications/Vagrant/incubator-metron-master/metron-deployment/target/rat.txt) before you can proceed. Ryan On Apr 22, 2017, at 6:03 PM, Mark de Rijk <[email protected]<mailto:[email protected]>> wrote: Hi, When I was running the mvn clean install –DskipTests command I got the following error and metron-deployment was marked as failed: Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project metron-deployment: Too many files with unapproved license: 1 See RAT report in: /Users/mark/Applications/Vagrant/incubator-metron-master/metron-deployment/target/rat.txt -> [Help 1] Then when I went through: vagrant destroy –f vagrant up I got the following error: Failed to execute goal on project metron-writer: Could not resolve dependencies for project org.apache.metron:metron-writer:jar:0.3.1: Failed to collect dependencies at org.apache.storm:storm-hdfs:jar:1.0.1.2.5.0.0-1245 -> org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.5.0.0-1245 -> org.mortbay.jetty:jetty:jar:6.1.26.hwx: Failed to read artifact descriptor for org.mortbay.jetty:jetty:jar:6.1.26.hwx: Could not transfer artifact org.mortbay.jetty:jetty:pom:6.1.26.hwx from/to repository.jboss.org<http://repository.jboss.org> (http://repository.jboss.org/nexus/content/groups/public/): Connection reset -> [Help 1]", "[ERROR] ", "[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.", "[ERROR] Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", "[ERROR] For more information about the errors and possible solutions, please read the following articles:", "[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException";, "[ERROR] ", "[ERROR] After correcting the problems, you can resume the build with the command", "[ERROR] mvn <goals> -rf :metron-writer"], "warnings": []} It looks to me it is trying to retrieve something from: http://repository.jboss.org/nexus/content/groups/public Regards, Mark de Rijk From: Nick Allen <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Saturday, 22April, 2017 at 23:39 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Failure installing Full-Dev Vagrant VM I've noticed this myself recently on master. Try running the following first, which will build and install all of the Metron libs in your local M2 cache (~/.m2/repository/org/apache/metron/). cd incubator-metron mvn clean install -DskipTests Then after that completes, try to deploy with Vagrant. cd incubator-metron/metron-deployment/vagrant/full-dev-platform vagrant destroy -f vagrant up Hope that works for you. On Sat, Apr 22, 2017 at 5:38 PM, Mark de Rijk <[email protected]<mailto:[email protected]>> wrote: Hi, Docker is running but got the following error after going through a vagrant destroy and vagrant up cycle: "[ERROR] Failed to execute goal on project metron-writer: Could not resolve dependencies for project org.apache.metron:metron-writer:jar:0.3.1: Failed to collect dependencies at org.apache.storm:storm-hdfs:jar:1.0.1.2.5.0.0-1245 -> org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.5.0.0-1245 -> org.mortbay.jetty:jetty:jar:6.1.26.hwx: Failed to read artifact descriptor for org.mortbay.jetty:jetty:jar:6.1.26.hwx: Could not transfer artifact org.mortbay.jetty:jetty:pom:6.1.26.hwx from/to repository.jboss.org<http://repository.jboss.org> (http://repository.jboss.org/nexus/content/groups/public/): Connection reset -> [Help 1]", "[ERROR] ", "[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.", "[ERROR] Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", "[ERROR] For more information about the errors and possible solutions, please read the following articles:", "[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException";, "[ERROR] ", "[ERROR] After correcting the problems, you can resume the build with the command", "[ERROR] mvn <goals> -rf :metron-writer"], "warnings" Regards, Mark de Rijk From: Nick Allen <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Saturday, 22April, 2017 at 21:46 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Failure installing Full-Dev Vagrant VM Are you sure you have the Docker service running? On Sat, Apr 22, 2017 at 4:40 PM, Mark de Rijk <[email protected]<mailto:[email protected]>> wrote: Hi, Docker wasn’t installed and after installing I have the following maven error: [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (docker-build) on project metron-rpm: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]", "[ERROR] ", "[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.", "[ERROR] Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", "[ERROR] For more information about the errors and possible solutions, please read the following articles:", "[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException";, "[ERROR] ", "[ERROR] After correcting the problems, you can resume the build with the command", "[ERROR] mvn <goals> -rf :metron-rpm"], "warnings": []} Regards, Mark de Rijk From: Nick Allen <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Saturday, 22April, 2017 at 21:23 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Failure installing Full-Dev Vagrant VM Do you have Docker running? It is a dependency that you need installed on your build machine that is not mentioned in the README. We have a JIRA open to get that in the README. On Sat, Apr 22, 2017 at 4:02 PM, Mark de Rijk <[email protected]<mailto:[email protected]>> wrote: Hi, I am trying to install the full-dev VM using the instructions on: https://github.com/apache/incubator-metron/tree/master/metron-deployment/vagrant/full-dev-platform When I get to the vagrant up it starts running through the ansible playbook and it fails on the following error: [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec (docker-build) on project metron-rpm: Command execution failed. Cannot run program \"docker\" (in directory \"/Users/mark/Applications/Vagrant/incubator-metron-master/metron-deployment/packaging/docker/rpm-docker\"): error=2, No such file or directory -> [Help 1]", "[ERROR] ", "[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.", " [ERROR] Re-run Maven using the -X switch to enable full debug logging.", "[ERROR] ", "[ERROR] For more information about the errors and possible solutions, please read the following articles:", "[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException";, "[ERROR] ", "[ERROR] After correcting the problems, you can resume the build with the command", "[ERROR] mvn <goals> -rf :metron-rpm"], "warnings": []} I really want to get the environment up and running this weekend so I can get familiar as soon as possible so I can start contributing to the project. If I made some error please do let me know. Regards, Mark de Rijk
