http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/README.md b/metron-deployment/vagrant/README.md deleted file mode 100644 index 6a86b6a..0000000 --- a/metron-deployment/vagrant/README.md +++ /dev/null @@ -1,21 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> -# Vagrant Deployment - -- Fast CAPA Test Platform -- Full Dev Platform
http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/.gitignore ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/.gitignore b/metron-deployment/vagrant/fastcapa-test-platform/.gitignore deleted file mode 100644 index a8b42eb..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/.gitignore +++ /dev/null @@ -1 +0,0 @@ -*.retry http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/README.md b/metron-deployment/vagrant/fastcapa-test-platform/README.md deleted file mode 100644 index 213d115..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/README.md +++ /dev/null @@ -1,138 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> -Fastcapa Test Environment -========================= - -Provides a test environment for the development and testing of Fastcapa. The environment is automatically validated after it is created to ensure that Fastcapa is behaving correctly. - -Two virtualized nodes are launched with Vagrant that can communicate with one another over a private network. -- The `source` node uses Metron's `pcap_replay` functionality to transmit raw network packet data over a private network. -- The `sink` node is running `fastcapa` and is capturing these network packets. -- Fastcapa then transforms and bundles the packets into a message. -- The message is sent to a Kafka broker running on the `source` node. - -Getting Started ---------------- - -The Fastcapa test environment can be executed on different operating systems. There is a sub-directory for each operating system that Fastcapa can be tested on. - -To run, simply execute `vagrant up` within the appropriate directory. For example, to run the tests on CentOS 7.1 then execute the following commands. -``` -cd centos-7.1 -vagrant up -``` - -Automated tests are executed after provisioning completes to ensure that Fastcapa and the rest of the environment is functioning properly. If you see something like the following, then the tests have passed. -``` -$ vagrant up -==> source: Running provisioner: ansible... - source: Running ansible-playbook... -... -TASK [debug] ******************************************************************* -ok: [source] => { - "msg": "Successfully received packets sent from pcap-replay!" -} -... -TASK [debug] ******************************************************************* -ok: [source] => { - "msg": "Successfully received a Kafka message from fastcapa!" -} -``` - -If the deployment process fails mid-course, running `vagrant provision` will continue the process from where it left off. This can sometimes occur when the VM reboots as part of the deployment process. The error might look like the following. -``` -TASK [fastcapa : Restart for modified kernel params] *************************** -fatal: [sink]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 127.0.0.1 closed.\r\n", "unreachable": true} - to retry, use: --limit @/Users/nallen/Development/metron/metron-deployment/vagrant/fastcapa-test-platform/playbook.retry - -PLAY RECAP ********************************************************************* -sink : ok=11 changed=9 unreachable=1 failed=0 -source : ok=29 changed=25 unreachable=0 failed=0 - -Ansible failed to complete successfully. Any error output should be -visible above. Please fix these errors and try again. -``` - -Going Deeper ------------- - -This section will outline in more detail the environment and how to interact with it. - -### `source` - -To validate that the `source` node is functioning properly, run the following commands. - -First, ensure that the `pcap-replay` service is running. - -``` -vagrant ssh source -sudo service pcap-replay status -``` - -Use `tcpdump` to ensure that the raw packet data is being sent over the private network. Enter 'CTRL-C' to kill the `tcpdump` process once you are able to see that packets are being sent. - -``` -sudo yum -y install tcpdump -sudo tcpdump -i enp0s8 -``` - -### `sink` - -Next validate that the `sink` is functioning properly. Run the following commands starting from the host operating system. - -First, ensure that the `fastcapa` service is running. - -``` -vagrant ssh sink -service fastcapa status -``` - -Ensure that the raw network packet data is being received by Kafka. - -``` -/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic pcap -``` - -Enter 'CTRL-C' to kill the `kafka-console-consumer` process once you are able to see that packets are being sent. These packets will appear to be gibberish in the console. This is the raw binary network packet data after all. - -FAQ ---- - -### Error Message: Timed out while waiting for the machine to boot - -``` -Timed out while waiting for the machine to boot. This means that -Vagrant was unable to communicate with the guest machine within -the configured ("config.vm.boot_timeout" value) time period. -If you look above, you should be able to see the error(s) that -Vagrant had when attempting to connect to the machine. These errors -are usually good hints as to what may be wrong. -If you're using a custom box, make sure that networking is properly -working and you're able to connect to the machine. It is a common -problem that networking isn't setup properly in these boxes. -Verify that authentication configurations are also setup properly, -as well. -If the box appears to be booting properly, you may want to increase -the timeout ("config.vm.boot_timeout") value. -â centos-7.4 git:(master) â vagrant status -Current machine states: -source running (virtualbox) -sink not created (virtualbox) -``` - -If you are unable to launch any of the Fastcapa test environments, which results in a message like the one above, then you may need to upgrade your version of Virtualbox. Success has been reported with versions of VirtualBox 5.1.22+. http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/Vagrantfile ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/Vagrantfile b/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/Vagrantfile deleted file mode 100644 index 179ca34..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/Vagrantfile +++ /dev/null @@ -1,72 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -Vagrant.configure("2") do |config| - - # enable hostmanager - config.hostmanager.enabled = true - config.hostmanager.manage_host = true - - # - # source - # - config.vm.define "source" do |node| - - # host settings - node.vm.hostname = "source" - node.vm.box = "bento/centos-7.1" - node.ssh.insert_key = "true" - node.vm.network :private_network, ip: "192.168.33.10", netmask: "255.255.255.0" - - # provider - node.vm.provider "virtualbox" do |vb| - vb.memory = 1024 - vb.cpus = 1 - end - end - - # - # sink - # - config.vm.define "sink" do |node| - - # host settings - node.vm.hostname = "sink" - node.vm.box = "bento/centos-7.1" - node.ssh.insert_key = "true" - node.vm.network :private_network, ip: "192.168.33.11", netmask: "255.255.255.0" - - # provider - node.vm.provider "virtualbox" do |vb| - vb.memory = 4096 - vb.cpus = 3 - - # network adapter settings; [Am79C970A|Am79C973|82540EM|82543GC|82545EM|virtio] - vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"] - vb.customize ["modifyvm", :id, "--nictype2","82545EM"] - end - - # provision host - node.vm.provision :ansible do |ansible| - ansible.limit = "all" - ansible.playbook = "../playbook.yml" - ansible.extra_vars = "vars/main.yml" - end - end - - -end http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/ansible.cfg ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/ansible.cfg b/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/ansible.cfg deleted file mode 100644 index 0a83e32..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/ansible.cfg +++ /dev/null @@ -1,22 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -[defaults] -host_key_checking = false -library = ../../../extra_modules -roles_path = ../../../roles -pipelining = True http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/vars/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/vars/main.yml b/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/vars/main.yml deleted file mode 100644 index 563fc2a..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.1/vars/main.yml +++ /dev/null @@ -1,52 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - -# 0.9.4+ required for fastcapa -librdkafka_version: 0.9.4 -librdkafka_url: https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz - -dpdk_device: ["00:08.0"] -dpdk_target: "x86_64-native-linuxapp-gcc" -num_huge_pages: 512 -kafka_broker_url: source:9092 -zookeeper_url: source:2181 -pcap_replay_interface: enp0s3 -kafka_broker_home: /usr/hdp/current/kafka-broker/ - -# fastcapa settings -fastcapa_portmask: 0x01 -fastcapa_kafka_config: /etc/fastcapa.conf -fastcapa_topic: pcap -fastcapa_burst_size: 32 -fastcapa_nb_rx_desc: 1024 -fastcapa_nb_rx_queue: 1 -fastcapa_tx_ring_size: 2048 - -# dummy variables for pycapa's dependence on ambari_gather_facts -cluster_name: dummy -namenode_host: dummy -core_site_tag: dummy -hdfs_url: dummy -kafka_broker_hosts: dummy -kafka_broker_tag: dummy -kafka_broker_port: dummy -zookeeper_hosts: dummy -zookeeper_tag: dummy -zookeeper_port: dummy -metron_hosts: dummy -kibana_hosts: dummy http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/Vagrantfile ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/Vagrantfile b/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/Vagrantfile deleted file mode 100644 index c62a0f1..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/Vagrantfile +++ /dev/null @@ -1,72 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -Vagrant.configure("2") do |config| - - # enable hostmanager - config.hostmanager.enabled = true - config.hostmanager.manage_host = true - - # - # source - # - config.vm.define "source" do |node| - - # host settings - node.vm.hostname = "source" - node.vm.box = "bento/centos-7.4" - node.ssh.insert_key = "true" - node.vm.network :private_network, ip: "192.168.33.10", netmask: "255.255.255.0" - - # provider - node.vm.provider "virtualbox" do |vb| - vb.memory = 1024 - vb.cpus = 1 - end - end - - # - # sink - # - config.vm.define "sink" do |node| - - # host settings - node.vm.hostname = "sink" - node.vm.box = "bento/centos-7.4" - node.ssh.insert_key = "true" - node.vm.network :private_network, ip: "192.168.33.11", netmask: "255.255.255.0" - - # provider - node.vm.provider "virtualbox" do |vb| - vb.memory = 4096 - vb.cpus = 3 - - # network adapter settings; [Am79C970A|Am79C973|82540EM|82543GC|82545EM|virtio] - vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"] - vb.customize ["modifyvm", :id, "--nictype2","82545EM"] - end - - # provision host - node.vm.provision :ansible do |ansible| - ansible.limit = "all" - ansible.playbook = "../playbook.yml" - ansible.extra_vars = "vars/main.yml" - end - end - - -end http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/ansible.cfg ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/ansible.cfg b/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/ansible.cfg deleted file mode 100644 index 0a83e32..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/ansible.cfg +++ /dev/null @@ -1,22 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -[defaults] -host_key_checking = false -library = ../../../extra_modules -roles_path = ../../../roles -pipelining = True http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/vars/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/vars/main.yml b/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/vars/main.yml deleted file mode 100644 index 563fc2a..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/centos-7.4/vars/main.yml +++ /dev/null @@ -1,52 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - -# 0.9.4+ required for fastcapa -librdkafka_version: 0.9.4 -librdkafka_url: https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz - -dpdk_device: ["00:08.0"] -dpdk_target: "x86_64-native-linuxapp-gcc" -num_huge_pages: 512 -kafka_broker_url: source:9092 -zookeeper_url: source:2181 -pcap_replay_interface: enp0s3 -kafka_broker_home: /usr/hdp/current/kafka-broker/ - -# fastcapa settings -fastcapa_portmask: 0x01 -fastcapa_kafka_config: /etc/fastcapa.conf -fastcapa_topic: pcap -fastcapa_burst_size: 32 -fastcapa_nb_rx_desc: 1024 -fastcapa_nb_rx_queue: 1 -fastcapa_tx_ring_size: 2048 - -# dummy variables for pycapa's dependence on ambari_gather_facts -cluster_name: dummy -namenode_host: dummy -core_site_tag: dummy -hdfs_url: dummy -kafka_broker_hosts: dummy -kafka_broker_tag: dummy -kafka_broker_port: dummy -zookeeper_hosts: dummy -zookeeper_tag: dummy -zookeeper_port: dummy -metron_hosts: dummy -kibana_hosts: dummy http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml b/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml deleted file mode 100644 index e038691..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/playbook.yml +++ /dev/null @@ -1,49 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -# -# the 'source' produces network traffic -# -- hosts: source - become: yes - roles: - - role: kafka-broker - - role: pcap_replay - - { role: pycapa, install_pycapa_service: False } - - { role: sensor-test-mode, pcap_replay: True, install_yaf: False, install_snort: False, install_pcap_replay: True } - tasks: - - service: name=pcap-replay state=started - -# -# the 'sink' consumes network traffic -# -- hosts: sink - become: yes - roles: - - role: librdkafka - - role: fastcapa - tasks: - - service: name=fastcapa state=started - -# -# validate the environment - needs to run on `source` node -# -- hosts: source - become: yes - tasks: - - include: tasks/validate-packets-sent.yml - - include: tasks/validate-messages-received.yml http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml b/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml deleted file mode 100644 index 5109a0e..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-messages-received.yml +++ /dev/null @@ -1,38 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- set_fact: - kafka_consumer_cmd: "{{ kafka_broker_home}}/bin/kafka-console-consumer.sh --zookeeper {{ zookeeper_url }} --topic {{ fastcapa_topic }} --max-messages 10 --from-beginning" - -- name: Attempting to receive message(s) sent by fastcapa - shell: "{{ kafka_consumer_cmd }}" - async: 30 - register: kafka_consumer - -- name: Waiting to receive message(s) sent by fastcapa - async_status: jid={{ kafka_consumer.ansible_job_id }} - register: job - until: job.finished - retries: 30 - -- fail: - msg: "No messages received from fastcapa within timeout: {{ kafka_consumer_cmd }}" - when: job.finished != 1 - -- debug: var=job - -- debug: msg="Successfully received a Kafka message from fastcapa!" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml b/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml deleted file mode 100644 index 25347dd..0000000 --- a/metron-deployment/vagrant/fastcapa-test-platform/tasks/validate-packets-sent.yml +++ /dev/null @@ -1,41 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install tcpdump - yum: name=tcpdump - -- set_fact: - sniff_cmd: "tcpdump -i {{ pcap_replay_interface }} -c 10" - -- name: "Attempting to sniff packet(s)" - shell: "{{ sniff_cmd }}" - async: 30 - register: sniffer - -- name: Waiting to receive packet(s) - async_status: jid={{ sniffer.ansible_job_id }} - register: job - until: job.finished - retries: 30 - -- fail: - msg: "No packets received from pcap-replay within timeout: {{ sniff_cmd }}" - when: job.finished != 1 - -- debug: var=job - -- debug: msg="Successfully received packets sent from pcap-replay!" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/full-dev-platform/.gitignore ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/.gitignore b/metron-deployment/vagrant/full-dev-platform/.gitignore deleted file mode 100644 index 8000dd9..0000000 --- a/metron-deployment/vagrant/full-dev-platform/.gitignore +++ /dev/null @@ -1 +0,0 @@ -.vagrant http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/full-dev-platform/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/README.md b/metron-deployment/vagrant/full-dev-platform/README.md deleted file mode 100644 index 4a58915..0000000 --- a/metron-deployment/vagrant/full-dev-platform/README.md +++ /dev/null @@ -1,123 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. ---> -Full Development Platform -========================= - -This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running on Virtualbox. - -Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. To that end the vagrant file configuration defaults to disabling solr and yaf. - -Getting Started ---------------- - -### Prerequisites - -The computer used to deploy Apache Metron will need to have the following components installed. - - - [Ansible](https://github.com/ansible/ansible) (2.0.0.2 or 2.2.2.0) - - [Docker](https://www.docker.com/community-edition) - - [Vagrant](https://www.vagrantup.com) 1.8+ - - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) - - [Virtualbox](https://virtualbox.org) 5.0+ - - Python 2.7 - - Maven 3.3.9 - - C++11 compliant compiler, like [GCC](https://gcc.gnu.org/projects/cxx-status.html#cxx11) - -Running the following script can help validate whether you have all the prerequisites installed and running correctly. - - ``` - metron-deployment/scripts/platform-info.sh - ``` - -#### How do I install these on MacOS? - -Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. - -1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/). - -1. Run the following command in a terminal to install all of the required tools. - - ``` - brew cask install vagrant virtualbox docker - brew cask install caskroom/versions/java8 - brew install maven@3.3 git - pip install ansible==2.2.2.0 - vagrant plugin install vagrant-hostmanager - open /Applications/Docker.app - ``` - -### Deploy Metron - -1. Ensure that the Docker service is running. - -1. Deploy Metron - - ``` - cd metron-deployment/vagrant/full-dev-platform - vagrant up - ``` - - Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host. - - ``` - vagrant provision - ``` - -### Explore Metron - -Navigate to the following resources to explore your newly minted Apache Metron environment. - - - [Ambari](http://node1:8080) - - [Metron](http://node1:5000) - - [Services](http://node1:2812) - -Connecting to the host through SSH is as simple as running the following command. - -``` -vagrant ssh -``` - -### Working with Metron - -In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. The following commands will re-run the `web` role on the Vagrant image. This will install components (if necessary) and start the UI. - -``` -./run_ansible_role.sh web -``` -or - -``` -vagrant --ansible-tags="web" provision -``` - -#### Using Tags - -A collection of tags is specified as a comma separated list. - -``` -./run_ansible_role.sh "sensors,enrichment" - -``` - -Tags are listed in the playbooks, some frequently used tags: -+ `hdp-install` - Install HDP -+ `hdp-deploy` - Deploy and Start HDP Services (will start all Hadoop Services) -+ `sensors` - Deploy and Start Sensors. -+ `enrichment` - Deploy and Start Enrichment Topology. - -Note also that there is a convenience script `./run_enrichment_role.sh` which executes Vagrant with the `enrichment` tag. http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/full-dev-platform/Vagrantfile ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/Vagrantfile b/metron-deployment/vagrant/full-dev-platform/Vagrantfile deleted file mode 100644 index 15df777..0000000 --- a/metron-deployment/vagrant/full-dev-platform/Vagrantfile +++ /dev/null @@ -1,92 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -require 'getoptlong' - -ansibleTags='' -ansibleSkipTags='sensors' - -begin - opts = GetoptLong.new( - [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ], - [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ] - ) - - opts.quiet = TRUE - - opts.each do |opt, arg| - case opt - when '--ansible-tags' - ansibleTags=arg - when '--ansible-skip-tags' - ansibleSkipTags=arg - end - end -rescue Exception => ignored -#Ignore to allow other opts to be passed to Vagrant -end - -puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != '' -puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != '' - -hosts = [{ - hostname: "node1", - ip: "192.168.66.121", - memory: "8192", - cpus: 4, - promisc: 2 # enables promisc on the 'Nth' network interface -}] - -Vagrant.configure(2) do |config| - - # all hosts built on centos 6 - config.vm.box = "metron/centos_base" - config.ssh.insert_key = true - - # enable the hostmanager plugin - config.hostmanager.enabled = true - config.hostmanager.manage_host = true - - # host definition - hosts.each_with_index do |host, index| - config.vm.define host[:hostname] do |node| - - # host settings - node.vm.hostname = host[:hostname] - node.vm.network "private_network", ip: host[:ip] - - # vm settings - node.vm.provider "virtualbox" do |vb| - vb.memory = host[:memory] - vb.cpus = host[:cpus] - - # enable promisc mode on the network interface - if host.has_key?(:promisc) - vb.customize ["modifyvm", :id, "--nicpromisc#{host[:promisc]}", "allow-all"] - end - end - end - end - - # provisioning - config.vm.provision :ansible do |ansible| - ansible.playbook = "../../playbooks/metron_full_install.yml" - ansible.sudo = true - ansible.tags = ansibleTags.split(",") if ansibleTags != '' - ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != '' - ansible.inventory_path = "../../inventory/full-dev-platform" - end -end http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/full-dev-platform/ansible.cfg ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/ansible.cfg b/metron-deployment/vagrant/full-dev-platform/ansible.cfg deleted file mode 100644 index 74f3fdf..0000000 --- a/metron-deployment/vagrant/full-dev-platform/ansible.cfg +++ /dev/null @@ -1,27 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -[defaults] -host_key_checking = false -library = ../../extra_modules -roles_path = ../../roles -pipelining = True -log_path = ./ansible.log - - -# fix for "ssh throws 'unix domain socket too long' " problem -[ssh_connection] -control_path = %(directory)s/%%h-%%p-%%r http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh b/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh deleted file mode 100755 index 0e5d831..0000000 --- a/metron-deployment/vagrant/full-dev-platform/run_ansible_role.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -vagrant --ansible-tags="$@" provision http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh b/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh deleted file mode 100755 index e26bc16..0000000 --- a/metron-deployment/vagrant/full-dev-platform/run_enrichment_role.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -./run_ansible_role.sh enrichment http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-interface/metron-rest/README.md ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md index 3d7d7a1..d51dc52 100644 --- a/metron-interface/metron-rest/README.md +++ b/metron-interface/metron-rest/README.md @@ -433,7 +433,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available * Returns: * 200 - Returns sample message * 404 - Either Kafka topic is missing or contains no messages - + ### `POST /api/v1/kafka/topic/{name}/produce` * Description: Produces a message to a Kafka topic * Input: @@ -442,7 +442,7 @@ Request and Response objects are JSON formatted. The JSON schemas are available * Returns: * 200 - Published * 404 - Kafka topic is missing - + ### `POST /api/v1/metaalert/searchByAlert` * Description: Get all meta alerts that contain an alert. * Input: @@ -846,7 +846,7 @@ The metron-rest application will be available at http://localhost:8080/swagger-u ### Full Dev -Start the [Full Dev](../../metron-deployment/vagrant/full-dev-platform) environment. Build the metron-rest module and start it with the Spring Boot Maven plugin: +Start the [development environment](../../metron-deployment/development/centos6). Build the metron-rest module and start it with the Spring Boot Maven plugin: ``` mvn clean package mvn spring-boot:run -Drun.profiles=vagrant,dev
