Hello Fellow Metronites:
As I explored my very first setup of Metron for the vagrant install, I ran into
a few hiccups.
I have listed below a summary of a rather long email with the details broken
out. Apologies for the lengthiness.
1. Hiccups while performing the pip install ansible step
2. install Xcode command line tools
3. install Xcode command line tools
4. After steps 1 and 2 were performed, I did a build from master and attempted
to startup the vagrant instance. Below is a dump of that attempts. Vagrant
still failed to come up. More fixes:
5. enable solr and disable elastic
6. make install verbose
7. ansible correction based on feedback that I'm using head of master and its
not in synch with the specified version of ansible on the page
1 Hiccups while performing the pip install ansible step:
```I was able to get this to work by upgrading openssl using brew
brew upgrade openssl
and then setting the CPPFLAGS and LDFLAGS to point to the openssl lib from brew
export CPPFLAGS=-I/usr/local/opt/openssl/include
export LDFLAGS=-L/usr/local/opt/openssl/lib```
when installing Vagrant at the step `pip install ansible==2.2.2.0`
i encountered problems with the crypt... python packages.
had to ensure that `brew upgrade openssl`
was performed.
2. install Xcode command line tools (again for some reason???)
`$ xcode-select --install`
3. errors that occurred while sensible installed with bad SSL brew casks. -
(see step 1):
```creating
build/temp.macosx-10.13-intel-2.7/build/temp.macosx-10.13-intel-2.7
cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os
-pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX
-DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os
-Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c
build/temp.macosx-10.13-intel-2.7/_openssl.c -o
build/temp.macosx-10.13-intel-2.7/build/temp.macosx-10.13-intel-2.7/_openssl.o
-Wconversion -Wno-error=sign-conversion
build/temp.macosx-10.13-intel-2.7/_openssl.c:498:10: fatal error:
'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
^~~~~~~~~~~~~~~~~~~~
1 error generated.
error: command 'cc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -c "import setuptools,
tokenize;__file__='/private/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-build-uWUxRG/cryptography/setup.py';exec(compile(getattr(tokenize,
'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
install --record
/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-OpX1S3-record/install-record.txt
--single-version-externally-managed --compile" failed with error code 1 in
/private/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-build-uWUxRG/cryptography```
4. After steps 1 and 2 were performed, I did a build from master and attempted
to startup the vagrant instance. Below is a dump of that attempts. Vagrant
still failed to come up. More fixes:
Tried to just use vagrant up and got a fail ....
This snippet has the following pieces of information to help you help me:
4.1. information about my machine/env
4.2. overview of sequence of commands to reproduce
4.3. terminal dump
Here are the details:
4.1. Information about my machine
4.1.1. unamme output:
~/workspace/metron/metron-deployment/development/centos6 (master) $ uname -a
Darwin ScottCotesLWMBP.fios-router.home 17.7.0 Darwin Kernel Version 17.7.0:
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
4.1.2. platform-info.sh output
~/workspace/metron/metron-deployment/scripts (master) $ ./platform-info.sh
Metron 0.6.1
--
* master
--
commit 0ba18026222f004316b96633780509c71934121e (HEAD -> master, origin/master,
origin/HEAD)
Author: JonZeolla <[email protected]>
Date: Mon Oct 22 19:48:56 2018 -0400
METRON-1826 Update librdkafka and devtoolset (JonZeolla via jonzeolla)
closes apache/metron#1238
--
metron-deployment/development/centos6/Vagrantfile | 1 +
metron-deployment/development/centos6/ansible/inventory/group_vars/all | 4 ++--
2 files changed, 3 insertions(+), 2 deletions(-)
--
ansible 2.2.2.0
config file =
configured module search path = Default w/o overrides
--
Vagrant 2.1.5
--
vagrant-hostmanager (1.8.9, global)
--
Python 2.7.10
--
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
2017-04-03T14:39:06-05:00)
Maven home: /usr/local/Cellar/maven/3.5.0/libexec
Java version: 1.8.0_181, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
--
Docker version 18.06.1-ce, build e68fc7a
--
node
v10.7.0
--
npm
6.2.0
--
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
--
Compiler is C++11 compliant
--
Darwin ScottCotesLWMBP.fios-router.home 17.7.0 Darwin Kernel Version 17.7.0:
Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
--
Total System Memory = 16384 MB
Processor Model: Intel(R) Core(TM) i7-4770HQ CPU
Processor Speed: 2.20GHz
Total Physical Processors: 4
Total cores: 4
Disk information:
/dev/disk1s1 466Gi 408Gi 50Gi 90% 3427183 9223372036851348624 0% /
/dev/disk1s4 466Gi 7.0Gi 50Gi 13% 7 9223372036854775800 0%
/private/var/vm
This CPU appears to support virtualization
4.2. overview of sequence of commands to reproduce
followed instructions for vagrant setup with the last following steps:
4.2.1. ~/workspace/metron/metron-deployment/development/centos6 (master) $
vagrant destroy -f
4.2.2. ~/workspace/metron/metron-deployment/development/centos6 (master) $ cd
../../..
4.2.3. ~/workspace/metron (master) $ git pull
4.2.4. ~/workspace/metron (master) $ mvn clean install -DskipTests -T C4
4.2.5. ~/workspace/metron (master) $ cd metron-deployment/development/centos6
4.2.4. ~/workspace/metron/metron-deployment/development/centos6 (master) $
vagrant up
4.3. terminal dump
...................................................
..--->>>redacted terminal dump for brevity<<<--- ..
...................................................
[INFO] --- maven-install-plugin:2.4:install (default-install) @ metron-rest ---
[INFO] Installing
/Users/scottcote/workspace/metron/metron-interface/metron-rest/target/metron-rest-0.6.1.jar
to
/Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1.jar
[INFO] Installing
/Users/scottcote/workspace/metron/metron-interface/metron-rest/pom.xml to
/Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1.pom
[INFO] Installing
/Users/scottcote/workspace/metron/metron-interface/metron-rest/target/metron-rest-0.6.1-archive.tar.gz
to
/Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1-archive.tar.gz
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Metron ............................................. SUCCESS [ 14.439 s]
[INFO] metron-stellar ..................................... SUCCESS [ 1.317 s]
[INFO] stellar-common ..................................... SUCCESS [ 39.920 s]
[INFO] metron-analytics ................................... SUCCESS [ 1.221 s]
[INFO] metron-maas-common ................................. SUCCESS [ 1.142 s]
[INFO] metron-platform .................................... SUCCESS [ 1.666 s]
[INFO] metron-zookeeper ................................... SUCCESS [ 5.004 s]
[INFO] metron-test-utilities .............................. SUCCESS [ 2.454 s]
[INFO] metron-integration-test ............................ SUCCESS [ 20.909 s]
[INFO] metron-maas-service ................................ SUCCESS [ 20.476 s]
[INFO] metron-common ...................................... SUCCESS [ 34.899 s]
[INFO] metron-statistics .................................. SUCCESS [ 33.713 s]
[INFO] metron-writer ...................................... SUCCESS [ 32.067 s]
[INFO] metron-storm-kafka-override ........................ SUCCESS [ 11.703 s]
[INFO] metron-storm-kafka ................................. SUCCESS [ 2.069 s]
[INFO] metron-hbase ....................................... SUCCESS [ 3.316 s]
[INFO] metron-profiler-common ............................. SUCCESS [ 2.298 s]
[INFO] metron-profiler-client ............................. SUCCESS [ 31.861 s]
[INFO] metron-profiler-storm .............................. SUCCESS [01:24 min]
[INFO] metron-profiler-spark .............................. SUCCESS [01:26 min]
[INFO] metron-profiler-repl ............................... SUCCESS [ 22.633 s]
[INFO] metron-hbase-client ................................ SUCCESS [ 23.168 s]
[INFO] metron-enrichment .................................. SUCCESS [01:27 min]
[INFO] metron-indexing .................................... SUCCESS [ 23.352 s]
[INFO] metron-solr ........................................ SUCCESS [01:06 min]
[INFO] metron-job ......................................... SUCCESS [ 1.818 s]
[INFO] metron-pcap ........................................ SUCCESS [ 4.094 s]
[INFO] metron-parsers ..................................... SUCCESS [01:14 min]
[INFO] metron-pcap-backend ................................ SUCCESS [ 54.775 s]
[INFO] metron-data-management ............................. SUCCESS [01:13 min]
[INFO] metron-management .................................. SUCCESS [ 12.919 s]
[INFO] elasticsearch-shaded ............................... SUCCESS [ 25.123 s]
[INFO] metron-elasticsearch ............................... SUCCESS [01:22 min]
[INFO] metron-deployment .................................. SUCCESS [ 1.265 s]
[INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 2.529 s]
[INFO] Metron Ambari Management Pack ...................... SUCCESS [ 3.869 s]
[INFO] metron-contrib ..................................... SUCCESS [ 1.151 s]
[INFO] metron-docker ...................................... SUCCESS [ 4.692 s]
[INFO] metron-performance ................................. SUCCESS [ 4.457 s]
[INFO] metron-interface ................................... SUCCESS [ 9.772 s]
[INFO] metron-config ...................................... SUCCESS [05:00 min]
[INFO] metron-alerts ...................................... SUCCESS [03:48 min]
[INFO] metron-rest-client ................................. SUCCESS [ 1.148 s]
[INFO] metron-rest ........................................ SUCCESS [ 33.988 s]
[INFO] site-book .......................................... SUCCESS [ 1.929 s]
[INFO] 3rd party Functions (just for tests) ............... SUCCESS [ 0.583 s]
[INFO] stellar-zeppelin ................................... SUCCESS [ 2.079 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:46 min (Wall Clock)
[INFO] Finished at: 2018-10-23T09:22:06-05:00
[INFO] Final Memory: 220M/2041M
[INFO] ------------------------------------------------------------------------
~/workspace/metron (master) $ cd metron-deployment/development/centos6/
~/workspace/metron/metron-deployment/development/centos6 (master) $ vagrant up
Running with ansible-skip-tags: ["sensors"]
Bringing machine 'node1' up with 'virtualbox' provider...
==> node1: Importing base box 'centos/6'...
==> node1: Matching MAC address for NAT networking...
==> node1: Checking if box 'centos/6' is up to date...
==> node1: Setting the name of the VM: centos6_node1_1540304760857_3427
==> node1: Clearing any previously set network interfaces...
==> node1: Preparing network interfaces based on configuration...
node1: Adapter 1: nat
node1: Adapter 2: hostonly
==> node1: Forwarding ports...
node1: 22 (guest) => 2222 (host) (adapter 1)
==> node1: Running 'pre-boot' VM customizations...
==> node1: Booting VM...
==> node1: Waiting for machine to boot. This may take a few minutes...
node1: SSH address: 127.0.0.1:2222
node1: SSH username: vagrant
node1: SSH auth method: private key
node1:
node1: Vagrant insecure key detected. Vagrant will automatically replace
node1: this with a newly generated keypair for better security.
node1:
node1: Inserting generated public key within guest...
node1: Removing insecure key from the guest if it's present...
node1: Key inserted! Disconnecting and reconnecting using new SSH key...
==> node1: Machine booted and ready!
==> node1: Checking for guest additions in VM...
node1: No guest additions were detected on the base box for this VM! Guest
node1: additions are required for forwarded ports, shared folders, host only
node1: networking, and more. If SSH fails on this machine, please install
node1: the guest additions and repackage the box to continue.
node1:
node1: This is not an error message; everything may continue to work
properly,
node1: in which case you may ignore this message.
==> node1: Setting hostname...
==> node1: Configuring and enabling network interfaces...
node1: SSH address: 127.0.0.1:2222
node1: SSH username: vagrant
node1: SSH auth method: private key
==> node1: Rsyncing folder:
/Users/scottcote/workspace/metron/metron-deployment/development/centos6/ =>
/vagrant
==> node1: [vagrant-hostmanager:guests] Updating hosts file on active guest
virtual machines...
==> node1: [vagrant-hostmanager:host] Updating hosts file on your workstation
(password may be required)...
Password:
==> node1: Running provisioner: ansible...
Vagrant has automatically selected the compatibility mode '2.0'
according to the Ansible version installed (2.2.2.0).
Alternatively, the compatibility mode can be specified in your Vagrantfile:
https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode
node1: Running ansible-playbook...
PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false
ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o
IdentityFile=/Users/scottcote/workspace/metron/metron-deployment/development/centos6/.vagrant/machines/node1/virtualbox/private_key
-o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --connection=ssh
--timeout=30 --extra-vars=ansible_user\=\'vagrant\' --limit="node1"
--inventory-file=ansible/inventory --become -vvv --skip-tags=sensors
ansible/playbook.yml
Using
/Users/scottcote/workspace/metron/metron-deployment/development/centos6/ansible.cfg
as config file
ERROR! no action detected in task. This often indicates a misspelled module
name, or incorrect module path.
The error appears to have been in
'/Users/scottcote/workspace/metron/metron-deployment/ansible/roles/enable-swap/tasks/main.yml':
line 29, column 3, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- include_tasks: enable-swap.yml
^ here
The error appears to have been in
'/Users/scottcote/workspace/metron/metron-deployment/ansible/roles/enable-swap/tasks/main.yml':
line 29, column 3, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- include_tasks: enable-swap.yml
^ here
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
5. enable solr and disable elastic
go to `ansible/inventory/group_vars/all`
and set
`# search
install_elasticsearch: False
install_solr: True
solr_collection_name: Metron
solr_number_shards: 1
solr_replication_factor: 1
elasticsearch_transport_port: 9300
## The elasticsearch_network_interface must be in the form specified for
Elasticsearch, with leading and trailing underscores.
elasticsearch_network_interface: _eth1_
elasticsearch_web_port: 9200
ansible/inventory/group_vars/all
`
6 make install verbose
mod this file
~/devprojects/metron/metron-deployment/development/centos6/Vagrantfile
er, metron/metron-deployment/development/centos6/Vagrantfile
under config.vm.provision add this
ansible.verbose = "vvv"
7. ansible correction based on feedback that I'm using head of master and its
not in synch with the specified version of ansible on the page
`pip install ansible==2.5.0`
then do `vagrant up`
