Hi Earl,
See attached. The document is in Markdown format. There's still a couple
of things which aren't working as expected like Kibana... I'll try to
keep the document updated with my findings.
-Laurens
I would like to get the directions u used to get your installation
running.
Sent with AquaMail for Android
http://www.aqua-mail.com
On May 8, 2017 3:08:48 PM Laurens Vets <[email protected]> wrote:
Hi list,
I'm not sure where to post this, but I've got a simple document which
explains installing Metron 0.4.0. I've been trying to install Metron
0.4.0 in 3 VMs the past couple of days and with the help of Ryan, Jon
&
Otto succeeded today.
I've got Metron 0.4.0 installed on CentOS 7 with a MariaDB database
for
the REST interface.
What or where would be the best place to post this?
[//]: # (Metron 0.4.0 bare metal installation guide for CentOS 7)
[//]: # (Written by Laurens Vets, [email protected])
[//]: # (Version 0.1, May 2017)
## Metron 0.4.0 with HDP 2.5 bare-metal install on Centos 7 with MariaDB for
Metron REST: ##
### Introduction ###
We will be installing Metron 0.4.0 with HDP 2.5 on CentOS 7. We will also
install MariaDB as a database for Metron REST. Additionally, we'll also install
the Metron Management UI and Apache NiFi.
I installed Metron in a test environment with 3 VMs to try it out, but this
guide can easily be adapted for other environments.
### Environment ###
- 3 VMs, 2 CPUs per VM and 8 GB RAM per VM.
- Hosts:
10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3
### Prerequisites: ###
- CentOS 7
- Set up passwordless SSH between our nodes:
If passwordless ssh has not yet been set up within the cluster, then in main
node generate key:
```
# cat /dev/zero | ssh-keygen -q -N "" 2>/dev/null
```
Add this newly generated key to all the slave nodes:
```
ssh-copy-id -i ~/.ssh/id_rsa.pub <replace_with_node_ip>
```
- Increase limits for ElasticSearch and Storm on nodes where you will be
installing them (if you don't know, increase it everywhere):
```
# echo -e "elasticsearch - memlock unlimited\nstorm - nproc 257597" >>
/etc/security/limits.conf
```
- Disable IPv6, leaving it enabled may force service to bind to IPv6 addresses
only and thus resulting in inability to connect to it ([source
link](https://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df)):
```
# sysctl -w net.ipv6.conf.all.disable_ipv6=1
# sysctl -w net.ipv6.conf.default.disable_ipv6=1
# echo -e "\n# Disable IPv6\nnet.ipv6.conf.all.disable_ipv6 =
1\nnet.ipv6.conf.default.disable_ipv6 = 1" >> /etc/sysctl.conf
```
- Disable Transparent Hugepage. Add "transparent_hugepage=never" to the end of
the kernel line in grub.conf and reboot. (Ambari demands it, do we need to
comply?):
```
transparent_hugepage=never
```
After reboot check that changes were applied (make sure that word "never" is
selected in square-brackets):
```
# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
```
### Metron install pre-preparation: ###
- On all nodes Install pre-requisites for Ambari:
```
# yum install epel-release -y
# yum update -y
# yum install git wget curl rpm tar unzip bzip2 wget createrepo yum-utils ntp
python-pip psutil install mariadb-server mysql-connector-java.jar -y
```
- On main node where your Ambari will be located install following:
```
# pip install --upgrade pip
# pip install --upgrade setuptools
```
- Install Maven on main node and on Metron node install java 1.8 (if you don't
know which it is, install it everywhere):
```
# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel -y
```
- Set path to Java 8 if it does not exist:
```
# export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s_/jre/bin/java__")
```
- Save export for future reboots:
```
# echo 'export JAVA_HOME=$(readlink -f /usr/bin/java | sed
"s_/jre/bin/java__")' > /etc/profile.d/java_18.sh
# chmod +x /etc/profile.d/java_18.sh
```
- Download and install Maven:
```
# wget
http://apache.volia.net/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz
# tar -zxf apache-maven-3.3.9-bin.tar.gz;
# mv apache-maven-3.3.9 /opt/export
# PATH=/opt/apache-maven-3.3.9/bin:$PATH
# echo 'export PATH=/opt/apache-maven-3.3.9/bin:$PATH' > /etc/profile.d/maven.sh
# chmod +x /etc/profile.d/maven.sh
```
- On Ambari node install and enable docker (we will need it to build Metron
mpack for Ambari):
```
# yum install docker-io -y
# service docker start
```
- Also on your build box, install npm. This is needed to build metron-config,
part of the UI.
```
# yum install npm -y
```
- Configure our database for Metron REST:
If you haven't run `mysql_secure_installation` after the database installation,
do that first:
```
[root@node1 ~]# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MySQL to secure it, we'll need the current
password for the root user. If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.
Set root password? [Y/n]
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n]
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n]
... Success!
By default, MySQL comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n]
- Dropping test database...
ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist
... Failed! Not critical, keep moving...
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n]
... Success!
All done! If you've completed all of the above steps, your MySQL
installation should now be secure.
Thanks for using MySQL!
Cleaning up...
[root@node1 ~]#
```
- Configure the Metron REST username and password:
```
# mysql -u root -p
CREATE USER '<DB USERNAME>'@'localhost' IDENTIFIED BY '<DB PASSWORD>';
CREATE DATABASE IF NOT EXISTS <DB NAME>;
GRANT ALL PRIVILEGES ON <DB NAME>.* TO '<DB USERNAME>'@'localhost';
```
For example:
```
# mysql -u root -p
CREATE USER 'metron'@'localhost' IDENTIFIED BY 'metron';
CREATE DATABASE IF NOT EXISTS metronrest;
GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'localhost';
```
Add the Metron REST username and password to the metronrest database:
```
# mysql -u metron -p
use metronrest;
insert into users (username, password, enabled) values
('<USERNAME>','<PASSWORD>',1);
insert into authorities (username, authority) values ('<USERNAME>',
'ROLE_USER');
```
For example, to use the username 'metron' with password 'metron', do the
following:
```
# mysql -u metron -p
use metronrest;
insert into users (username, password, enabled) values ('metron','metron',1);
insert into authorities (username, authority) values ('metron', 'ROLE_USER');
```
### Build Metron code ###
Now we are going to start to building Metron. At the time of writing, Metron
0.4.0 was in the final stages of being released.
- On the main node, clone the Metron repository:
```
# git clone https://github.com/apache/incubator-metron
```
- Build Metron with HDP 2.5 profile:
```
# cd incubator-metron
# mvn clean install -DskipTests -PHDP-2.5.0.0
# cd metron-deployment/packaging/docker/rpm-docker
# mvn clean install -DskipTests -PHDP-2.5.0.0
```
- On all nodes, create a localrepo directory and copy the RPMs from Ambari node
there:
```
# mkdir /localrepo
# cp -rp
/root/incubator-metron/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch/*
/localrepo/
# createrepo /localrepo
# scp scp /localrepo/* <replace_with_node_ip>:/localrepo/
```
- Make sure to run `createrepo /localrepo` on every node!
### Ambari 2.4 with HDP 2.5 install ###
Inspired by:
[http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-installation/content/ch_Getting_Ready.html]
- Adjust limits to secure level (inspired by
[link](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/ref-729d1fb0-6d1b-459f-a18a-b5eba4540ab5.1.html)):
```
# ulimit -n 32768
# ulimit -u 65536
# echo -e "* - nofile 32768\n* - nproc 65536" >> /etc/security/limits.conf
```
- Enable time sync, disable firewall and SElinux:
```
# chkconfig ntpd on
# service ntpd start
# chkconfig iptables off
# /etc/init.d/iptables stop
# setenforce 0 (=> I know, but for the sake of simplicity, quickness & testing,
I've disabled selinux.)
```
- Make sure each node can resolve every other node's hostname or add hostname
of each node to `/etc/hosts` on every node. For example add following lines in
/etc/hosts of each node:
10.10.10.1 node1
10.10.10.2 node2
10.10.10.3 node3
Where 10.10.10.1, 10.10.10.2 and 10.10.10.3 are the IPs of your nodes and
node1, node2 and node3 are hostnames.
- On main node download and setup Ambari repo (you may replace the "2.4.2.0"
with a newer Ambari version number):
```
# wget -nv
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari.repo
-O /etc/yum.repos.d/ambari.repo
```
- Check that it was added:
```
# yum repolist | grep ambari
Updates-ambari-2.4.2.0 ambari-2.4.2.0 - Updates 12
```
- Install and setup Ambari server:
```
# yum install ambari-server -y
# ambari-server setup -s
```
- Add Metron service to Ambari by running mpack command (make sure to specify
correct path to mpack in --mpack=):
```
# ambari-server install-mpack
--mpack=incubator-metron/metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-0.4.0.0.tar.gz
--verbose
```
- Start Ambari:
```
# ambari-server start
```
- Access the Ambari UI by going to the following URL in a web browser:
`http://<replace_with_master_node_ip>:8080/#/installer/step0`. You can use
admin/admin as username/password.
**Get Started page:** Enter any desired cluster name.
**Select Version:** Make sure "Public Repository" is checked. You should also
see the `/localrepo` directory listed.
**Install Options**: Specify hostnames of your nodes where Ambari cluster
should be installed (all the ones you have specified in /etc/hosts) in Target
Hosts. Copy content of the main node private key (/root/.ssh/id_rsa) in "Host
Registration Information".
If you receive warning like below, ignore it and click OK:
"The following hostnames are not valid FQDNs"
**Choose Services:** Select following Services:
Metron related:
HDFS
YARN + MapReduce2
HBase
Zookeeper
Storm
Flume
Kafka
Elasticsearch
Kibana
Metron
Ambari Metrics
Dependencies:
Pig
Tez
Slider
Ambari will probably select additional services as well.
**Assign Masters:** Assign "Kafka Broker" on all nodes. Make sure move
following components on one common node (Taken from previous guide, is this
still necessary?):
Storm UI Server
Metron Indexing
MySQL Server
Kibana Server
Elasticsearch Master
Metron Parsers
Metron Enrichment
**Assign Slaves and Clients:** select All for:
DataNode
NodeManager
RegionServer
Supervisor
Client
**Customize Services:** Following is a list of services that need to be
configured:
- For ElasticSearch:
* Set "zen_discovery_ping_unicast_hosts" to the IP of the node where you
assigned ElasticSearch Master on the Assign Master tab.
* Under "Advanced elastic-site": Change "network_host" to "0.0.0.0". Do not
do this if your Metron is exposed to the public internet!
- Kibana:
Set "kibana_es_url" to
`http://<replace_with_elasticsearch_master_hostname>:9200`.
"replace_with_elasticsearch_master_hostname" is the IP of the node where you
assigned ElasticSearch Master on the Assign Master tab.
- Metron:
Set "Elasticsearch Hosts" to the IP of the node where you assigned
ElasticSearch Master on the Assign Master tab.
- For metron REST use:
Metron JDBC URL: jdbc:mysql://127.0.0.1:3306/<DB NAME>
Metron JDBC client path: /usr/share/java/mysql-connector-java.jar
Metron JDBC Driver: com.mysql.jdbc.Driver
Metron JDBC password: <DB PASSWORD>
Metron JDBC platform: mysql
Metron JDBC username: <DB USERNAME>
- Set rest of the configuration values to recommended by Ambari or the ones you
desire (like DB passwords) and perform install. In the end, I ended up with:
node1 | node2 | node3
-------------------------|-------------------------|-------------------------
DataNode | App Timeline Server | DataNode
Elasticsearch Master | DataNode | Elasticsearch Data Node
HBase Client | DRPC Server | Flume
HBase Master | HBase Client | HBase Client
RegionServer | RegionServer | RegionServer
HCat Client | HCat Client | HCat Client
HDFS Client | HDFS Client | HDFS Client
Hive Client | History Server | Hive Client
Kafka Broker | Hive Client | Kafka Broker
Kibana Server | Hive Metastore | MapReduce2 Client
MapReduce2 Client | HiveServer2 | Metrics Collector
Grafana | Kafka Broker | Metrics Monitor
Metrics Monitor | MapReduce2 Client | Metron Client
Metron Client | Metrics Monitor | NodeManager
Metron Enrichment | Metron Client | Pig Client
Metron Indexing | MySQL Server | Slider Client
Metron Parsers | Nimbus | Spark Client
Metron REST | NodeManager | Supervisor
NameNode | Pig Client | Tez Client
NodeManager | ResourceManager | YARN Client
Pig Client | SNameNode | ZooKeeper Client
Slider Client | Slider Client | ZooKeeper Server
Spark Client | Spark Client |
Spark History Server | Supervisor |
Storm UI Server | Tez Client
Supervisor | WebHCat Server
Tez Client | YARN Client
YARN Client | ZooKeeper Client
Zeppelin Notebook | ZooKeeper Server
ZooKeeper Client |
ZooKeeper Server |
- There's 1 last step we need to do before all services will run.
Due to systemd in Centos 7, doing `service metron-rest start <PASSWORD>` no
longer works. Instead we have to edit the configuration file
"`/etc/rc.d/init.d/metron-rest`". In this file, change
`METRON_JDBC_PASSWORD="$2"` to `METRON_JDBC_PASSWORD="<DB PASSWORD>"` and
restart the metron-rest service via the Ambari interface or via CLI:
```
# systemctl start metron-rest.service
```
### Miscellaneous ###
- Install Apache NiFi. Download nifi-1.2.0-bin.tar.gz from
https://nifi.apache.org/download.html
```
# wget <URL>
# tar xf nifi-1.2.0-bin.tar.gz
```
Before we run NiFi, we need to change the port as the default port collides
with the Ambari port. To do this, we need to change the value
"nifi.web.http.port=8080" to "nifi.web.http.port=8089" in the file
"nifi-1.1.2/conf/nifi.properties". Install and start NiFi afterwards:
```
# nifi-1.1.2/bin/nifi.sh install
# nifi-1.1.2/bin/nifi.sh start
```
- Install Metron Management UI:
<If metron REST is on other machine copy file over there!>
```
cd /root/incubator-metron/
cp metron-interface/metron-config/target/metron-config-0.4.0-archive.tar.gz
/usr/metron/0.4.0/
cd /usr/metron/0.4.0/
tar -xf metron-config-0.4.0-archive.tar.gz
npm install [email protected] -g
export METRON_REST_URL=http://127.0.0.1:8082
./bin/start_management_ui.sh -p 4200 -r http://127.0.0.1:8082
```
- Install monit (To be done.)
### Exposed Interfaces ##
In the end, you'll end up with a bunch of exposed UIs:
- Ambari: http://metron1:8080/
- Kibana: http://metron1:5000/
- Storm UI: http://metron1:8744/
- Metron REST: http://metron1:8082/swagger-ui.html#/
- Management UI: http://node1:4200/ (user/password)
- Apache Nifi: http://metron1:8089/nifi/
