Thanks for the feedback Scott. I see 3 categories: 1. Some setup and prereq doc clarification for newcomers, e.g. need for xcode install, clarify doc version compatibility 2. Added details for configuring Solr instead of ES 3. Some added details around self-help on debugging the install, e.g. enabling vvv with ansible (I was actually the dev that worked through that with Ward), some suggestion about finding errors in Ambari.
Some of this could probably be simple Ansible enhancements. Have I missed anything? Are you still in the middle of getting full dev up or is it working for you now? Best, Mike Miklavcic On Oct 24, 2018 4:03 PM, "Scott Cote" <[email protected]> wrote: Hello Fellow Metronites: As I explored my very first setup of Metron for the vagrant install, I ran into a few hiccups. I have listed below a summary of a rather long email with the details broken out. Apologies for the lengthiness. 1. Hiccups while performing the pip install ansible step 2. install Xcode command line tools 3. install Xcode command line tools 4. After steps 1 and 2 were performed, I did a build from master and attempted to startup the vagrant instance. Below is a dump of that attempts. Vagrant still failed to come up. More fixes: 5. enable solr and disable elastic 6. make install verbose 7. ansible correction based on feedback that I'm using head of master and its not in synch with the specified version of ansible on the page 1 Hiccups while performing the pip install ansible step: ```I was able to get this to work by upgrading openssl using brew brew upgrade openssl and then setting the CPPFLAGS and LDFLAGS to point to the openssl lib from brew export CPPFLAGS=-I/usr/local/opt/openssl/include export LDFLAGS=-L/usr/local/opt/openssl/lib``` when installing Vagrant at the step `pip install ansible==2.2.2.0` i encountered problems with the crypt... python packages. had to ensure that `brew upgrade openssl` was performed. 2. install Xcode command line tools (again for some reason???) `$ xcode-select --install` 3. errors that occurred while sensible installed with bad SSL brew casks. - (see step 1): ```creating build/temp.macosx-10.13-intel-2.7/build/temp.macosx-10.13-intel-2.7 cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c build/temp.macosx-10.13-intel-2.7/_openssl.c -o build/temp.macosx-10.13-intel-2.7/build/temp.macosx-10.13-intel-2.7/_openssl.o -Wconversion -Wno-error=sign-conversion build/temp.macosx-10.13-intel-2.7/_openssl.c:498:10: fatal error: 'openssl/opensslv.h' file not found #include <openssl/opensslv.h> ^~~~~~~~~~~~~~~~~~~~ 1 error generated. error: command 'cc' failed with exit status 1 ---------------------------------------- Command "/usr/bin/python -c "import setuptools, tokenize;__file__='/private/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-build-uWUxRG/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-OpX1S3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-build-uWUxRG/cryptography``` 4. After steps 1 and 2 were performed, I did a build from master and attempted to startup the vagrant instance. Below is a dump of that attempts. Vagrant still failed to come up. More fixes: Tried to just use vagrant up and got a fail .... This snippet has the following pieces of information to help you help me: 4.1. information about my machine/env 4.2. overview of sequence of commands to reproduce 4.3. terminal dump Here are the details: 4.1. Information about my machine 4.1.1. unamme output: ~/workspace/metron/metron-deployment/development/centos6 (master) $ uname -a Darwin ScottCotesLWMBP.fios-router.home 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 4.1.2. platform-info.sh output ~/workspace/metron/metron-deployment/scripts (master) $ ./platform-info.sh Metron 0.6.1 -- * master -- commit 0ba18026222f004316b96633780509c71934121e (HEAD -> master, origin/master, origin/HEAD) Author: JonZeolla <[email protected]> Date: Mon Oct 22 19:48:56 2018 -0400 METRON-1826 Update librdkafka and devtoolset (JonZeolla via jonzeolla) closes apache/metron#1238 -- metron-deployment/development/centos6/Vagrantfile | 1 + metron-deployment/development/centos6/ansible/inventory/group_vars/all | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) -- ansible 2.2.2.0 config file = configured module search path = Default w/o overrides -- Vagrant 2.1.5 -- vagrant-hostmanager (1.8.9, global) -- Python 2.7.10 -- Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T14:39:06-05:00) Maven home: /usr/local/Cellar/maven/3.5.0/libexec Java version: 1.8.0_181, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" -- Docker version 18.06.1-ce, build e68fc7a -- node v10.7.0 -- npm 6.2.0 -- Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 10.0.0 (clang-1000.11.45.2) Target: x86_64-apple-darwin17.7.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin -- Compiler is C++11 compliant -- Darwin ScottCotesLWMBP.fios-router.home 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 -- Total System Memory = 16384 MB Processor Model: Intel(R) Core(TM) i7-4770HQ CPU Processor Speed: 2.20GHz Total Physical Processors: 4 Total cores: 4 Disk information: /dev/disk1s1 466Gi 408Gi 50Gi 90% 3427183 9223372036851348624 0% / /dev/disk1s4 466Gi 7.0Gi 50Gi 13% 7 9223372036854775800 0% /private/var/vm This CPU appears to support virtualization 4.2. overview of sequence of commands to reproduce followed instructions for vagrant setup with the last following steps: 4.2.1. ~/workspace/metron/metron-deployment/development/centos6 (master) $ vagrant destroy -f 4.2.2. ~/workspace/metron/metron-deployment/development/centos6 (master) $ cd ../../.. 4.2.3. ~/workspace/metron (master) $ git pull 4.2.4. ~/workspace/metron (master) $ mvn clean install -DskipTests -T C4 4.2.5. ~/workspace/metron (master) $ cd metron-deployment/development/centos6 4.2.4. ~/workspace/metron/metron-deployment/development/centos6 (master) $ vagrant up 4.3. terminal dump ................................................... ..--->>>redacted terminal dump for brevity<<<--- .. ................................................... [INFO] --- maven-install-plugin:2.4:install (default-install) @ metron-rest --- [INFO] Installing /Users/scottcote/workspace/metron/metron-interface/metron-rest/target/metron-rest-0.6.1.jar to /Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1.jar [INFO] Installing /Users/scottcote/workspace/metron/metron-interface/metron-rest/pom.xml to /Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1.pom [INFO] Installing /Users/scottcote/workspace/metron/metron-interface/metron-rest/target/metron-rest-0.6.1-archive.tar.gz to /Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1-archive.tar.gz [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Metron ............................................. SUCCESS [ 14.439 s] [INFO] metron-stellar ..................................... SUCCESS [ 1.317 s] [INFO] stellar-common ..................................... SUCCESS [ 39.920 s] [INFO] metron-analytics ................................... SUCCESS [ 1.221 s] [INFO] metron-maas-common ................................. SUCCESS [ 1.142 s] [INFO] metron-platform .................................... SUCCESS [ 1.666 s] [INFO] metron-zookeeper ................................... SUCCESS [ 5.004 s] [INFO] metron-test-utilities .............................. SUCCESS [ 2.454 s] [INFO] metron-integration-test ............................ SUCCESS [ 20.909 s] [INFO] metron-maas-service ................................ SUCCESS [ 20.476 s] [INFO] metron-common ...................................... SUCCESS [ 34.899 s] [INFO] metron-statistics .................................. SUCCESS [ 33.713 s] [INFO] metron-writer ...................................... SUCCESS [ 32.067 s] [INFO] metron-storm-kafka-override ........................ SUCCESS [ 11.703 s] [INFO] metron-storm-kafka ................................. SUCCESS [ 2.069 s] [INFO] metron-hbase ....................................... SUCCESS [ 3.316 s] [INFO] metron-profiler-common ............................. SUCCESS [ 2.298 s] [INFO] metron-profiler-client ............................. SUCCESS [ 31.861 s] [INFO] metron-profiler-storm .............................. SUCCESS [01:24 min] [INFO] metron-profiler-spark .............................. SUCCESS [01:26 min] [INFO] metron-profiler-repl ............................... SUCCESS [ 22.633 s] [INFO] metron-hbase-client ................................ SUCCESS [ 23.168 s] [INFO] metron-enrichment .................................. SUCCESS [01:27 min] [INFO] metron-indexing .................................... SUCCESS [ 23.352 s] [INFO] metron-solr ........................................ SUCCESS [01:06 min] [INFO] metron-job ......................................... SUCCESS [ 1.818 s] [INFO] metron-pcap ........................................ SUCCESS [ 4.094 s] [INFO] metron-parsers ..................................... SUCCESS [01:14 min] [INFO] metron-pcap-backend ................................ SUCCESS [ 54.775 s] [INFO] metron-data-management ............................. SUCCESS [01:13 min] [INFO] metron-management .................................. SUCCESS [ 12.919 s] [INFO] elasticsearch-shaded ............................... SUCCESS [ 25.123 s] [INFO] metron-elasticsearch ............................... SUCCESS [01:22 min] [INFO] metron-deployment .................................. SUCCESS [ 1.265 s] [INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 2.529 s] [INFO] Metron Ambari Management Pack ...................... SUCCESS [ 3.869 s] [INFO] metron-contrib ..................................... SUCCESS [ 1.151 s] [INFO] metron-docker ...................................... SUCCESS [ 4.692 s] [INFO] metron-performance ................................. SUCCESS [ 4.457 s] [INFO] metron-interface ................................... SUCCESS [ 9.772 s] [INFO] metron-config ...................................... SUCCESS [05:00 min] [INFO] metron-alerts ...................................... SUCCESS [03:48 min] [INFO] metron-rest-client ................................. SUCCESS [ 1.148 s] [INFO] metron-rest ........................................ SUCCESS [ 33.988 s] [INFO] site-book .......................................... SUCCESS [ 1.929 s] [INFO] 3rd party Functions (just for tests) ............... SUCCESS [ 0.583 s] [INFO] stellar-zeppelin ................................... SUCCESS [ 2.079 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:46 min (Wall Clock) [INFO] Finished at: 2018-10-23T09:22:06-05:00 [INFO] Final Memory: 220M/2041M [INFO] ------------------------------------------------------------------------ ~/workspace/metron (master) $ cd metron-deployment/development/centos6/ ~/workspace/metron/metron-deployment/development/centos6 (master) $ vagrant up Running with ansible-skip-tags: ["sensors"] Bringing machine 'node1' up with 'virtualbox' provider... ==> node1: Importing base box 'centos/6'... ==> node1: Matching MAC address for NAT networking... ==> node1: Checking if box 'centos/6' is up to date... ==> node1: Setting the name of the VM: centos6_node1_1540304760857_3427 ==> node1: Clearing any previously set network interfaces... ==> node1: Preparing network interfaces based on configuration... node1: Adapter 1: nat node1: Adapter 2: hostonly ==> node1: Forwarding ports... node1: 22 (guest) => 2222 (host) (adapter 1) ==> node1: Running 'pre-boot' VM customizations... ==> node1: Booting VM... ==> node1: Waiting for machine to boot. This may take a few minutes... node1: SSH address: 127.0.0.1:2222 node1: SSH username: vagrant node1: SSH auth method: private key node1: node1: Vagrant insecure key detected. Vagrant will automatically replace node1: this with a newly generated keypair for better security. node1: node1: Inserting generated public key within guest... node1: Removing insecure key from the guest if it's present... node1: Key inserted! Disconnecting and reconnecting using new SSH key... ==> node1: Machine booted and ready! ==> node1: Checking for guest additions in VM... node1: No guest additions were detected on the base box for this VM! Guest node1: additions are required for forwarded ports, shared folders, host only node1: networking, and more. If SSH fails on this machine, please install node1: the guest additions and repackage the box to continue. node1: node1: This is not an error message; everything may continue to work properly, node1: in which case you may ignore this message. ==> node1: Setting hostname... ==> node1: Configuring and enabling network interfaces... node1: SSH address: 127.0.0.1:2222 node1: SSH username: vagrant node1: SSH auth method: private key ==> node1: Rsyncing folder: /Users/scottcote/workspace/metron/metron-deployment/development/centos6/ => /vagrant ==> node1: [vagrant-hostmanager:guests] Updating hosts file on active guest virtual machines... ==> node1: [vagrant-hostmanager:host] Updating hosts file on your workstation (password may be required)... Password: ==> node1: Running provisioner: ansible... Vagrant has automatically selected the compatibility mode '2.0' according to the Ansible version installed (2.2.2.0). Alternatively, the compatibility mode can be specified in your Vagrantfile: https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode node1: Running ansible-playbook... PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o IdentityFile=/Users/scottcote/workspace/metron/metron-deployment/development/centos6/.vagrant/machines/node1/virtualbox/private_key -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --connection=ssh --timeout=30 --extra-vars=ansible_user\=\'vagrant\' --limit="node1" --inventory-file=ansible/inventory --become -vvv --skip-tags=sensors ansible/playbook.yml Using /Users/scottcote/workspace/metron/metron-deployment/development/centos6/ansible.cfg as config file ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path. The error appears to have been in '/Users/scottcote/workspace/metron/metron-deployment/ansible/roles/enable-swap/tasks/main.yml': line 29, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - include_tasks: enable-swap.yml ^ here The error appears to have been in '/Users/scottcote/workspace/metron/metron-deployment/ansible/roles/enable-swap/tasks/main.yml': line 29, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - include_tasks: enable-swap.yml ^ here Ansible failed to complete successfully. Any error output should be visible above. Please fix these errors and try again. 5. enable solr and disable elastic go to `ansible/inventory/group_vars/all` and set `# search install_elasticsearch: False install_solr: True solr_collection_name: Metron solr_number_shards: 1 solr_replication_factor: 1 elasticsearch_transport_port: 9300 ## The elasticsearch_network_interface must be in the form specified for Elasticsearch, with leading and trailing underscores. elasticsearch_network_interface: _eth1_ elasticsearch_web_port: 9200 ansible/inventory/group_vars/all ` 6 make install verbose mod this file ~/devprojects/metron/metron-deployment/development/centos6/Vagrantfile er, metron/metron-deployment/development/centos6/Vagrantfile under config.vm.provision add this ansible.verbose = "vvv" 7. ansible correction based on feedback that I'm using head of master and its not in synch with the specified version of ansible on the page `pip install ansible==2.5.0` then do `vagrant up`
