[
https://issues.apache.org/jira/browse/METRON-427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509868#comment-15509868
]
ASF GitHub Bot commented on METRON-427:
---
GitHub user justinleet opened a pull request:
https://github.com/apache/incubator-metron/pull/266
METRON-427 Create Ambari Management Pack for Metron Installation
This is an initial cut at an MVP for the Ambari Management Pack. Most of
the ground work was completed in other Jiras (detailed in METRON-427
description). This takes that work, adds the enrichment portion similiar to
indexing, elasticsearch, etc., and bundles it in an mpack.
This includes end to end from creating appropriate Kafka topics, preloading
a MySQL database with GeoIP data for enrichment, and making it available to an
Elasticsearch and Kibana instance. ES is optional, and can be redirected to an
outside instance.
The directory structuring changed to reflect the mpack MVP, and accordingly
a lot of files got moved/renamed. For most of the ground work, there are
various fixes and tweaks for consistency to each other and to fix bugs found
during testing.
Detailed instructions are found in the metron-deployment README.md in the
'Ambari Management Pack' section. Essentially these steps are:
- Build the RPMs per the previously existing instructions (make sure to run
the main build first).
- Build the mpack using the metron-deployment POM. From metron-deployment
just run `mvn clean package -DskipTests`. This will produce an mpack tar.gz,
which is what will be used in the installation to the Ambari server.
- Place the RPMs in /localrepo on the nodes. We don't have remote RPMs
yet, so this is a necessary evil that will go away.
- Place the mpack file on the ambari-server node.
- Install the mpack with `ambari-server` install-mpack
--mpack= --verbose`
- Update Storm's configuration in Ambari for topology.classpath to
/etc/hbase/conf:/etc/hadoop/conf. Ideally this will be automated after this
MVP.
- Run through the normal Ambari install process. There are some caveats
about this (colocation of Metron services isn't enforced, and Storm + Kafka +
HBase client need to be available). More details are in the README, but just
make sure Storm + Kafka + HBase client are available and that you aren't
installing to a node with MySQL already installed. The MySQL issue is most
likely on a single node install
- Use the action to install Elasticsearch templates where you can start /
stop services.
This was tested on an AWS cluster, and on a set of Docker instances where a
lot of components (e.g. MapReduce) disabled to avoid memory issues.
Steps used on Docker were (the server and agent images already have most of
the unnecessary components disabled. Make sure to change jleet to your user
home:
`docker-machine rm default
docker-machine create --driver virtualbox --virtualbox-disk-size "51200"
--virtualbox-memory "8192" default
docker-machine start default
eval $(docker-machine env default)
git clone https://github.com/sequenceiq/docker-ambari
cd docker-ambari
. ambari-functions
export AMBARI_SERVER_IMAGE=dlyle65535/ambari-server:2.4.0.1-jdk8
export AMBARI_AGENT_IMAGE=dlyle65535/ambari-agent:2.4.0.1-jdk8
export DOCKER_OPTS="-v /Users/jleet:/home/host -v
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch:/localrepo
-v
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/:/mpack"
(mac only) sudo route add -net 172.17.0.0/16 $(docker-machine ip default)
amb-start-cluster 5
amb-shell
blueprint add --url
https://raw.githubusercontent.com/dlyle65535/ambari-rest-client/docker-metron-test/src/main/resources/blueprints/docker-metron
cluster build --blueprint docker-metron
cluster assign --host amb1.service.consul --hostGroup master_1
cluster assign --host amb2.service.consul --hostGroup master_2
cluster assign --host amb3.service.consul --hostGroup master_3
cluster assign --host amb4.service.consul --hostGroup master_4
cluster create
docker exec -it amb-server bash
ambari-server install-mpack
--mpack=/mpack/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
ambari-server restart
http://amb-server.service.consul:8080 - install Metron service on
amb4.service.consul
es_url: can be invalid for test purpose otherwise, use authentic
One can docker it - docker run -d -p 9200:9200 -p 9300:9300
elasticsearch -Des.cluster.name=metron
Storm UI Server: amb1.service.consul:8744
`
Limitations of the MVP (also detailed in README)
- MySQL install should be optional (and allow for using an
