GitHub user dlyle65535 opened a pull request:
https://github.com/apache/incubator-metron/pull/241
METRON-400 Deploy Probes to running Docker Container
This was tested in the following ways:
Regression Testing ##
[X] Full Dev - Worked as expected
[X] EC2 - Worked as expected
Functional Testing ##
[X] Run on docker HDP instance from
[docker-ambari](https://github.com/sequenceiq/docker-ambari) Note: used custom
(dlyle65535/ambari-agent:2.4.0.0-1130-jdk8 and
dlyle65535/ambari-server:2.4.0.0-1130-jdk8)
Steps to Stand-Alone Test ##
*Note:*
You'll need docker.py installed and Ansible > 2, we still recommend 2.0.0.2.
Ansible will error without docker.py installed and give you a chance to:
```
pip install 'docker-py>=1.7.0'
```
Provision Container ###
Start the container:
```
docker run -d --hostname amb-server --privileged --name amb-server -it
dlyle65535/ambari-server:2.4.0.0-1130-jdk8
```
Run a shell on the container:
```
docker exec -it amb-server bash
```
Change nameserver to 8.8.8.8 (change to nameserver 8.8.8.8, delete 2nd line)
```
vi /etc/resolv.conf
```
Get HDP repo:
```
wget -nv
http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.2.0/hdp.repo
-O /etc/yum.repos.d/hdp.repo
```
Install Zookeeper:
```
yum install -y zookeeper-server
```
Setup and Start Zookeeper:
```
export ZOOKEEPER_CONF_DIR=/etc/zookeeper/conf
export ZOOKEEPER_HOME=/usr/hdp/current/zookeeper-server
export ZOO_LOG_DIR=/var/log/zookeeper
export ZOOPIDFILE=/var/run/zookeeper/zookeeper_server.pid
export SERVER_JVMFLAGS=-Xmx1024m
export JAVA=$JAVA_HOME/bin/java
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_HOME/*
export ZOOCFGDIR=$ZOOKEEPER_CONF_DIR
export ZOOCFG=zoo.cfg
source $ZOOKEEPER_CONF_DIR/zookeeper-env.sh
/usr/hdp/current/zookeeper-server/bin/zkServer.sh start
```
Test Zookeeper:
```
/usr/hdp/current/zookeeper-server/bin/zkCli.sh -server localhost:2181 ls /
```
The command should return:
> Connecting to localhost:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zookeeper]
>
Install and Start Kafka:
```
yum install -y kafka
/usr/hdp/current/kafka-broker/bin/kafka start
```
Test Kafka:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
localhost:2181 --create --topic test --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list
localhost:9092 --topic test
```
Add some test data and hit ctrl-c.
```
/usr/hdp/current/kafka-broker//bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --topic test --from-beginning
```
You should see your test data - ctrl-c to exit.
Create Probe Data Topics:
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
localhost:2181 --create --topic bro --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
localhost:2181 --create --topic snort --replication-factor 1 --partitions 1
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper
localhost:2181 --create --topic yaf --replication-factor 1 --partitions 1
```
Exit Container Shell:
```
exit
```
Install Probes using Ansible:
1. cd <metron-home>/metron-deployment/playbooks
2. In playbooks/docker_probe_install.yml change kafka_broker_url to
amb-server:9092 (kafka_broker_url: amb-server:9092)
3. export DOCKER_VERSION=<docker -version> e.g. 1.12.1 (omit any trailing
rc stuff)
ansible-playbook docker_probe_install.yml
4. run ansible-playbook
```
ansible-playbook docker_probe_install.yml
```
It should complete with:
> PLAY RECAP
*********************************************************************
> amb-server : ok=100 changed=60 unreachable=0 failed=0
> localhost : ok=2 changed=0 unreachable=0 failed=0
> docker exec -it amb-server bash
Start Sensor Probes:
```
service pcap-replay start
/usr/local/bro/bin/broctl start
service yaf start eth0
service snortd start
/usr/hdp/current/flume-server/bin/flume-ng agent -f
/etc/flume/conf/flume-snort.conf -n snort > /dev/null 2>&1 &
```
Check for Data
```
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --from-beginning --topic bro
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --from-beginning --topic snort
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper
localhost:2181 --from-beginning --topic yaf
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dlyle65535/incubator-metron
docker-probes-master-merge
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/241.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #241
----
commit f1f89271a426358c1ae45b079b1bfcdc48742db3
Author: David Lyle <[email protected]>
Date: 2016-08-31T16:29:53Z
METRON-400 Deploy Probes to running Docker Container
commit 26d0561748da28d3a6e562334816dd6ec04d4e46
Author: David Lyle <[email protected]>
Date: 2016-09-04T17:41:03Z
Only use amb4 for Kafka broker
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
