GitHub user justinleet opened a pull request:

    https://github.com/apache/incubator-metron/pull/266

    METRON-427 Create Ambari Management Pack for Metron Installation

    This is an initial cut at an MVP for the Ambari Management Pack.  Most of 
the ground work was completed in other Jiras (detailed in METRON-427 
description).  This takes that work, adds the enrichment portion similiar to 
indexing, elasticsearch, etc., and bundles it in an mpack.
    
    This includes end to end from creating appropriate Kafka topics, preloading 
a MySQL database with GeoIP data for enrichment, and making it available to an 
Elasticsearch and Kibana instance. ES is optional, and can be redirected to an 
outside instance.
    
    The directory structuring changed to reflect the mpack MVP, and accordingly 
a lot of files got moved/renamed.  For most of the ground work, there are 
various fixes and tweaks for consistency to each other and to fix bugs found 
during testing.
    
    Detailed instructions are found in the metron-deployment README.md in the 
'Ambari Management Pack' section.  Essentially these steps are:
    - Build the RPMs per the previously existing instructions (make sure to run 
the main build first).
    - Build the mpack using the metron-deployment POM. From metron-deployment 
just run `mvn clean package -DskipTests`.  This will produce an mpack tar.gz, 
which is what will be used in the installation to the Ambari server.
    - Place the RPMs in /localrepo on the nodes.  We don't have remote RPMs 
yet, so this is a necessary evil that will go away.
    - Place the mpack file on the ambari-server node.
    - Install the mpack with `ambari-server` install-mpack 
--mpack=<mpack_location> --verbose`
    - Update Storm's configuration in Ambari for topology.classpath to 
/etc/hbase/conf:/etc/hadoop/conf.  Ideally this will be automated after this 
MVP.
    - Run through the normal Ambari install process.  There are some caveats 
about this (colocation of Metron services isn't enforced, and Storm + Kafka + 
HBase client need to be available).  More details are in the README, but just 
make sure Storm + Kafka + HBase client are available and that you aren't 
installing to a node with MySQL already installed.  The MySQL issue is most 
likely on a single node install
    - Use the action to install Elasticsearch templates where you can start / 
stop services.
    
    This was tested on an AWS cluster, and on a set of Docker instances where a 
lot of components (e.g. MapReduce) disabled to avoid memory issues.
    
    Steps used on Docker were (the server and agent images already have most of 
the unnecessary components disabled.  Make sure to change jleet to your user 
home:
    `docker-machine rm default
    docker-machine create --driver virtualbox --virtualbox-disk-size "51200" 
--virtualbox-memory "8192" default
    
    docker-machine start default
    eval $(docker-machine env default)
    git clone https://github.com/sequenceiq/docker-ambari
    cd docker-ambari
    . ambari-functions
    export AMBARI_SERVER_IMAGE=dlyle65535/ambari-server:2.4.0.1-jdk8
    export AMBARI_AGENT_IMAGE=dlyle65535/ambari-agent:2.4.0.1-jdk8
    
    export DOCKER_OPTS="-v /Users/jleet:/home/host -v 
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch:/localrepo
 -v 
/Users/jleet/Documents/workspace/incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/:/mpack"
    
    (mac only) sudo route add -net 172.17.0.0/16 $(docker-machine ip default)
    
    amb-start-cluster 5
    amb-shell
        blueprint add --url 
https://raw.githubusercontent.com/dlyle65535/ambari-rest-client/docker-metron-test/src/main/resources/blueprints/docker-metron
        cluster build --blueprint docker-metron
        cluster assign --host amb1.service.consul --hostGroup master_1
        cluster assign --host amb2.service.consul --hostGroup master_2
        cluster assign --host amb3.service.consul --hostGroup master_3
        cluster assign --host amb4.service.consul --hostGroup master_4
        cluster create
    <WAIT FOR CLUSTER CREATION TO COMPLETE AND ^C TO EXIT>
    
    
    docker exec -it amb-server bash
    ambari-server install-mpack 
--mpack=/mpack/metron_mpack-1.0.0.0-SNAPSHOT.tar.gz --verbose
    ambari-server restart
    http://amb-server.service.consul:8080 - install Metron service on 
amb4.service.consul
        es_url: can be invalid for test purpose otherwise, use authentic 
                   One can docker it - docker run -d -p 9200:9200 -p 9300:9300 
elasticsearch -Des.cluster.name=metron
        Storm UI Server: amb1.service.consul:8744
    `
    
    
    
    
    Limitations of the MVP (also detailed in README)
    - MySQL install should be optional (and allow for using an existing 
instance).
    - MySQL should not be installed on a node already running a MySQL instance 
(e.g. an Ambari Server using MySQL as its database).
    - There is currently no hosting for RPMs remotely.  They will have to be 
built locally.
    - Colocation of appropriate services should be enforced by Ambari.  See 
[#Installing Management Pack] for more details.
    - Storm's topology.classpath is not updated with the Metron service install 
and needs to be updated separately.
    - Several configuration parameters used when installing the Metron service 
could (and should) be grabbed from Ambari.  Install will require them to be 
manually entered.
    - Need to handle upgrading Metron
    - Different versions of CentOS apparently have different named for MySQL, 
so we may have to adjust there.
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/justinleet/incubator-metron mpack

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/266.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #266
    
----
commit d4b8ea8c148f1753932f799e3c553847cdb3b05c
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-05T13:23:12Z

    sample

commit 0193f467f6b79a81479d30798525d981757d3146
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-09T22:14:10Z

    add parser and common-services

commit 67096076717087126bdcf1e07d5e28d5b55a52e7
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-10T22:12:04Z

    finish commands. refactor

commit 09dc18b9266702e90f4a7f02ea33edac31be0317
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-12T16:00:13Z

    fixes

commit b447901c078c2b943608e2b5304fb87fb2aaabad
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-12T16:18:18Z

    config fixes

commit 724070de4696059b98a358d585ff724b2f2226f9
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-16T14:10:27Z

    incremental progress. quicklinks. kafka home

commit e83964b800875ea341407e58b921f2429d0c517d
Author: David Lyle <dlyle65...@gmail.com>
Date:   2016-08-16T20:39:43Z

    Initial working version with mpack build and separate profile (build-rpms) 
for building rpms.

commit edd7b60c0565ef725858aa32e299c1b20d1f481b
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-16T21:01:46Z

    incremental fixes

commit 3172d9137b05f3f4c271f2a121fa4223606c8221
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-08-17T12:14:36Z

    WIP Indexing Service

commit 955680b20c6684c30631257a1068791a52e92a0a
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-17T21:09:44Z

    check service configured on start. fix zk load. fix topology start

commit 310550291051f7d649f8f4a51a80ab722758837c
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-18T20:21:33Z

    service check

commit 2a308376fb57ceaf2c875c0f0fb754c372d74f6f
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-18T21:40:12Z

    trying to fix service check

commit e9abe68bba0b1b95302c99a29f5fb872dd60f1c0
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-19T15:25:07Z

    cleanup/refactor. fix service check ordering

commit 8c1a093807c2ee18a6c9f315510bce2c2d0baf69
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-19T16:07:02Z

    fix service check

commit ffae457b34e94f5fdbf22a638c62bf69d032057a
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-19T21:34:14Z

    METRON-383 Ambari install for Metron Parser topologies.

commit d147e24097a76032b64f705456e6f3cfc335bff3
Author: justinjleet <justinjl...@gmail.com>
Date:   2016-08-17T12:50:06Z

    Metron Indexing Service in Ambari

commit ae7a62af1ce17abc86897f26d05c4fb751095134
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-22T15:40:48Z

    remove errant file

commit eb21e3651ec9f2a94f853684d94dd1d49fbbee2b
Author: David Lyle <dlyle65...@gmail.com>
Date:   2016-08-25T14:46:35Z

    Merge branch 'master' into AMBARI_MANAGEMENTPACK

commit 3d6e8faae5f3903986f51d162960dd3dae63f0f9
Author: David Lyle <dlyle65...@gmail.com>
Date:   2016-08-25T17:34:19Z

    Working ES and Kibana install.

commit ebe5539a4923da38f3285573bd4f26ed9a6f6ae8
Author: David Lyle <dlyle65...@gmail.com>
Date:   2016-08-25T17:38:26Z

    Merge branch 'METRON-383' of https://github.com/mmiklavc/incubator-metron 
into AMBARI_MANAGEMENTPACK

commit 32f7c63d17b0d62012d0e542cddfb9e81b0606c1
Author: David Lyle <dlyle65...@gmail.com>
Date:   2016-08-25T18:19:50Z

    Working parser integration

commit 8ab5c7b7d73cb6f98bb6957bb53e4ebfaaeab04d
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-05T13:23:12Z

    sample

commit 2a6f97830f72151d3813ac0a41d04ef31410fe87
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-09T22:14:10Z

    add parser and common-services

commit c2735510f1949b3682a0ecb8473d791c56e71315
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-10T22:12:04Z

    finish commands. refactor

commit 959991b6a0497acd93422da4c5cc8b09a43cd48c
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-12T16:00:13Z

    fixes

commit fe7f46822104d6b313ce80516916dcde9a0add4e
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-12T16:18:18Z

    config fixes

commit 8bab89166952c736962fe07a7478829575a5fb61
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-16T14:10:27Z

    incremental progress. quicklinks. kafka home

commit 43b6d04a8712c7332dd45116113b29ef61ebf8f9
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-16T21:01:46Z

    incremental fixes

commit 800322dc948cffbaf03788efc8cf756d8b8cb8cf
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-17T21:09:44Z

    check service configured on start. fix zk load. fix topology start

commit 0a723a458154e87f94210756cc3e2e49d3a805ad
Author: Michael Miklavcic <michael.miklav...@gmail.com>
Date:   2016-08-18T20:21:33Z

    service check

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Reply via email to