Hello, I have made the following pull request for two new ways to deploy to AWS. https://github.com/apache/metron/pull/916
i) AWS Single Node Cluster Deployment Using Vagrant I was able to deploy Metron as a Single node to AWS using the latest Metron code in git as of 2018-01-30 8AM UTC. However, after a while Ambari shows "Metron REST" as red and had to restart a few components. See pic for what Kibana looks like. ii) AWS Single Node Cluster Deployment Using an AMI In addition to including instructions for deploying 0.4.3 using Vagrant I also included instructions for deploying to AWS using a preexisting AMI (I made the following AMI public as a proof-of-concept: "GCR-Xetron Demo"/"ami-93cb4ff7"). This might be the quickest option for deployment since no pre-install requirements would be needed. However, Ambari shows a few components as red and had to restart them. -Ahmed _______________________________________________________________ Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR - Cybersecurity Operations Center Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> ________________________________ From: Ahmed Shah <ahmeds...@cmail.carleton.ca> Sent: January 25, 2018 2:17 PM To: Otto Fowler; u...@metron.apache.org Subject: Ability to deploy metron full dev as a single node to aws with vagrant Thanks, The old script worked for 0.4.1, but it seems to fail with 0.4.2. (I put the 0.4.2 code I'm working on here for now: https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/aws-vagrant/Vagrantfile ) It fails with 0.4.2 here: ansible.log 2018-01-25 13:36:38,321 p=10334 u=csocadmin | TASK [ambari_gather_facts : Ask Ambari: namenode_host] ************************* 2018-01-25 13:36:38,742 p=10334 u=csocadmin | ^[[0;31mfatal: [node1]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to resolve the host name given."}^[[0m 2018-01-25 13:36:38,752 p=10334 u=csocadmin | PLAY *************************************************************************** 2018-01-25 13:36:38,795 p=10334 u=csocadmin | TASK [setup] ******************************************************************* 2018-01-25 13:36:39,263 p=10334 u=csocadmin | ^[[0;32mok: [node1]^[[0m 2018-01-25 13:36:39,267 p=10334 u=csocadmin | TASK [epel : Install EPEL repository] ****************************************** . . . 2018-01-25 13:36:50,369 p=10334 u=csocadmin | TASK [ambari_gather_facts : set_fact] ****************************************** 2018-01-25 13:36:50,400 p=10334 u=csocadmin | ^[[0;36mskipping: [node1]^[[0m 2018-01-25 13:36:50,404 p=10334 u=csocadmin | TASK [ambari_gather_facts : Ask Ambari: namenode_host] ************************* 2018-01-25 13:36:50,872 p=10334 u=csocadmin | ^[[0;31mfatal: [node1]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to resolve the host name given."}^[[0m 2018-01-25 13:36:50,882 p=10334 u=csocadmin | PLAY *************************************************************************** 2018-01-25 13:36:50,942 p=10334 u=csocadmin | TASK [setup] ******************************************************************* 2018-01-25 13:36:51,482 p=10334 u=csocadmin | ^[[0;32mok: [node1]^[[0m 2018-01-25 13:36:51,486 p=10334 u=csocadmin | TASK [epel : Install EPEL repository] ****************************************** 2018-01-25 13:37:01,684 p=10334 u=csocadmin | ^[[0;32mok: [node1]^[[0m 2018-01-25 13:37:01,687 p=10334 u=csocadmin | TASK [python-pip : Install python-pip] ***************************************** 2018-01-25 13:37:02,301 p=10334 u=csocadmin | ^[[0;32mok: [node1]^[[0m 2018-01-25 13:37:02,304 p=10334 u=csocadmin | TASK [httplib2 : Install python httplib2 dependency] *************************** 2018-01-25 13:37:03,013 p=10334 u=csocadmin | ^[[0;32mok: [node1]^[[0m 2018-01-25 13:37:03,017 p=10334 u=csocadmin | TASK [ambari_gather_facts : Ask Ambari: cluster_name] ************************** 2018-01-25 13:37:03,086 p=10334 u=csocadmin | ^[[0;36mskipping: [node1]^[[0m 2018-01-25 13:37:03,089 p=10334 u=csocadmin | TASK [ambari_gather_facts : set_fact] ****************************************** 2018-01-25 13:37:03,147 p=10334 u=csocadmin | ^[[0;36mskipping: [node1]^[[0m 2018-01-25 13:37:03,150 p=10334 u=csocadmin | TASK [ambari_gather_facts : Ask Ambari: namenode_host] ************************* 2018-01-25 13:37:03,814 p=10334 u=csocadmin | ^[[0;31mfatal: [node1]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to resolve the host name given."}^[[0m 2018-01-25 13:37:03,815 p=10334 u=csocadmin | PLAY RECAP ********************************************************************* 2018-01-25 13:37:03,815 p=10334 u=csocadmin | 127.0.0.1 : ok=0 changed=0 unreachable=1 failed=0 2018-01-25 13:37:03,816 p=10334 u=csocadmin | node1 : ok=103 changed=48 unreachable=0 failed=5 When it works for 0.4.2 I will make the pull request into Metron. I'm running the vagrant file on a Mac. The /etc/hosts was updated to point node1 to the elastic_ip in the Mac. Any idea what the issue might be? -Ahmed _______________________________________________________________ Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR - Cybersecurity Operations Center Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> ________________________________ From: Otto Fowler <ottobackwa...@gmail.com> Sent: January 10, 2018 10:27 AM To: Ahmed Shah; u...@metron.apache.org Subject: Re: Intro & Question So, what would work would be: 1. Create a jira like “Ability to deploy metron full dev to aws with vagrant” With a description of the use case, and how the vagrant file will fill it. 2. create a pr, with the new file in metron-deployment/vagrant/aws 3. update the readme I think On January 10, 2018 at 10:16:03, Ahmed Shah (ahmeds...@cmail.carleton.ca<mailto:ahmeds...@cmail.carleton.ca>) wrote: Would be glad to. Where in github should I put it? -Ahmed _______________________________________________________________ Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR - Cybersecurity Operations Center Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> ________________________________ From: Otto Fowler <ottobackwa...@gmail.com<mailto:ottobackwa...@gmail.com>> Sent: January 9, 2018 11:51 AM To: Ahmed Shah; u...@metron.apache.org<mailto:u...@metron.apache.org> Subject: Re: Intro & Question Any interest in submitting this? On January 9, 2018 at 10:42:08, Ahmed Shah (ahmeds...@cmail.carleton.ca<mailto:ahmeds...@cmail.carleton.ca>) wrote: Hello Srikanth, Our team adapted the Metron 0.4.1 Single Node VM install (Original Code Here: https://github.com/apache/metron/tree/master/metron-deployment/vagrant/full-dev-platform) to deploy a single node to AWS. Our Vagrent file is here: https://github.com/LTW-GCR-CSOC/csoc-installation-scripts/blob/master/amazon-deploy/Metron/Vagrantfile You can define your AWS Elastic IP, Subnet ID, VPC, and Security Group ID before running the file. Hope it helps. -Ahmed _______________________________________________________________ Ahmed Shah (PMP, M. Eng.) Cybersecurity Analyst & Developer GCR - Cybersecurity Operations Center Carleton University - cugcr.com<https://cugcr.com/tiki/lce/index.php> ________________________________ From: Srikanth Nagarajan <s...@gandivanetworks.com<mailto:s...@gandivanetworks.com>> Sent: January 9, 2018 2:39 AM To: u...@metron.apache.org<mailto:u...@metron.apache.org> Subject: Intro & Question Hi My name is Srikanth and work for a Cyber Security firm. We are building Metron to test in our lab environment using AWS. 1. Is there a single VM version for Cloud install available ? If yes, please share procedure. 2. During the Amazon-Ec2 install for the multi node version provided in the metron git-hub docs https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2 get an error [WARNING]: * Failed to parse /Users/sri/metron/metron-deployment/amazon-ec2/ec2.py with script plugin: Inventory script (/Users/sri/metron/metron-deployment/amazon-ec2/ec2.py) had an execution error: ERROR: "Forbidden", while: getting ElastiCache clusters Any assistance would be appreciated. Thanks Srikanth ______________________ Srikanth Nagarajan Principal Gandiva Networks Inc 732.690.1884 Mobile s...@gandivanetworks.com<mailto:s...@gandivanetworks.com> www.gandivanetworks.com<http://www.gandivanetworks.com/> Please consider the environment before printing this. NOTICE: The information contained in this e-mail message is intended for addressee(s) only. If you have received this message in error please notify the sender.
