Thank you for your help now and earlier with Ward :) The talk at Activate went off very smooth - am looking forward to digging in.
Your summary => I think that was the gist of it. Not sure if its a mac problem or not, but don't leave out the brew upgrade of openssl and the setting of those environment variables. Maybe they were only a problem for ansible 2.2.2.0 Let me know if you want me to review document changes - it would help me better understand ya'lls documentation style. Kinda an easy way to start learning how to be a contributor. We had a whole talk at the activate conference on how to be a committer and documentation was one of the talking points. SCott > On Oct 24, 2018, at 5:48 PM, Michael Miklavcic <[email protected]> > wrote: > > Thanks for the feedback Scott. I see 3 categories: > 1. Some setup and prereq doc clarification for newcomers, e.g. need for xcode > install, clarify doc version compatibility > 2. Added details for configuring Solr instead of ES > 3. Some added details around self-help on debugging the install, e.g. > enabling vvv with ansible (I was actually the dev that worked through that > with Ward), some suggestion about finding errors in Ambari. > > Some of this could probably be simple Ansible enhancements. Have I missed > anything? Are you still in the middle of getting full dev up or is it working > for you now? > > Best, > Mike Miklavcic > > On Oct 24, 2018 4:03 PM, "Scott Cote" <[email protected] > <mailto:[email protected]>> wrote: > Hello Fellow Metronites: > > As I explored my very first setup of Metron for the vagrant install, I ran > into a few hiccups. > > I have listed below a summary of a rather long email with the details broken > out. Apologies for the lengthiness. > > 1. Hiccups while performing the pip install ansible step > 2. install Xcode command line tools > 3. install Xcode command line tools > 4. After steps 1 and 2 were performed, I did a build from master and > attempted to startup the vagrant instance. Below is a dump of that attempts. > Vagrant still failed to come up. More fixes: > 5. enable solr and disable elastic > 6. make install verbose > 7. ansible correction based on feedback that I'm using head of master and its > not in synch with the specified version of ansible on the page > > 1 Hiccups while performing the pip install ansible step: > > ```I was able to get this to work by upgrading openssl using brew > > brew upgrade openssl > > and then setting the CPPFLAGS and LDFLAGS to point to the openssl lib from > brew > > export CPPFLAGS=-I/usr/local/opt/openssl/include > export LDFLAGS=-L/usr/local/opt/openssl/lib``` > > when installing Vagrant at the step `pip install ansible==2.2.2.0` > i encountered problems with the crypt... python packages. > had to ensure that `brew upgrade openssl` > was performed. > > > 2. install Xcode command line tools (again for some reason???) > > `$ xcode-select --install` > > 3. errors that occurred while sensible installed with bad SSL brew casks. - > (see step 1): > ```creating > build/temp.macosx-10.13-intel-2.7/build/temp.macosx-10.13-intel-2.7 > cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g > -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX > -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os > -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch i386 -arch x86_64 -pipe > -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 > -c build/temp.macosx-10.13-intel-2.7/_openssl.c -o > build/temp.macosx-10.13-intel-2.7/build/temp.macosx-10.13-intel-2.7/_openssl.o > -Wconversion -Wno-error=sign-conversion > build/temp.macosx-10.13-intel-2.7/_openssl.c:498:10: fatal error: > 'openssl/opensslv.h' file not found > #include <openssl/opensslv.h> > ^~~~~~~~~~~~~~~~~~~~ > 1 error generated. > error: command 'cc' failed with exit status 1 > > ---------------------------------------- > Command "/usr/bin/python -c "import setuptools, > tokenize;__file__='/private/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-build-uWUxRG/cryptography/setup.py';exec(compile(getattr(tokenize, > 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" > install --record > /var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-OpX1S3-record/install-record.txt > --single-version-externally-managed --compile" failed with error code 1 in > /private/var/folders/vx/n3g3g7k94lg8pxz1hdp4bhb40000gp/T/pip-build-uWUxRG/cryptography``` > > > 4. After steps 1 and 2 were performed, I did a build from master and > attempted to startup the vagrant instance. Below is a dump of that attempts. > Vagrant still failed to come up. More fixes: > > Tried to just use vagrant up and got a fail .... > This snippet has the following pieces of information to help you help me: > 4.1. information about my machine/env > 4.2. overview of sequence of commands to reproduce > 4.3. terminal dump > Here are the details: > 4.1. Information about my machine > 4.1.1. unamme output: > ~/workspace/metron/metron-deployment/development/centos6 (master) $ uname -a > Darwin ScottCotesLWMBP.fios-router.home 17.7.0 Darwin Kernel Version 17.7.0: > Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 > 4.1.2. platform-info.sh output > ~/workspace/metron/metron-deployment/scripts (master) $ ./platform-info.sh > Metron 0.6.1 > -- > * master > -- > commit 0ba18026222f004316b96633780509c71934121e (HEAD -> master, > origin/master, origin/HEAD) > Author: JonZeolla <[email protected] <mailto:[email protected]>> > Date: Mon Oct 22 19:48:56 2018 -0400 > METRON-1826 Update librdkafka and devtoolset (JonZeolla via jonzeolla) > closes apache/metron#1238 > -- > metron-deployment/development/centos6/Vagrantfile | 1 + > metron-deployment/development/centos6/ansible/inventory/group_vars/all | 4 > ++-- > 2 files changed, 3 insertions(+), 2 deletions(-) > -- > ansible 2.2.2.0 > config file = > configured module search path = Default w/o overrides > -- > Vagrant 2.1.5 > -- > vagrant-hostmanager (1.8.9, global) > -- > Python 2.7.10 > -- > Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; > 2017-04-03T14:39:06-05:00) > Maven home: /usr/local/Cellar/maven/3.5.0/libexec > Java version: 1.8.0_181, vendor: Oracle Corporation > Java home: > /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac" > -- > Docker version 18.06.1-ce, build e68fc7a > -- > node > v10.7.0 > -- > npm > 6.2.0 > -- > Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr > --with-gxx-include-dir=/usr/include/c++/4.2.1 > Apple LLVM version 10.0.0 (clang-1000.11.45.2) > Target: x86_64-apple-darwin17.7.0 > Thread model: posix > InstalledDir: > /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin > -- > Compiler is C++11 compliant > -- > Darwin ScottCotesLWMBP.fios-router.home 17.7.0 Darwin Kernel Version 17.7.0: > Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64 > -- > Total System Memory = 16384 MB > Processor Model: Intel(R) Core(TM) i7-4770HQ CPU > Processor Speed: 2.20GHz > Total Physical Processors: 4 > Total cores: 4 > Disk information: > /dev/disk1s1 466Gi 408Gi 50Gi 90% 3427183 9223372036851348624 0% > / > /dev/disk1s4 466Gi 7.0Gi 50Gi 13% 7 9223372036854775800 0% > /private/var/vm > This CPU appears to support virtualization > > > 4.2. overview of sequence of commands to reproduce > followed instructions for vagrant setup with the last following steps: > 4.2.1. ~/workspace/metron/metron-deployment/development/centos6 (master) $ > vagrant destroy -f > 4.2.2. ~/workspace/metron/metron-deployment/development/centos6 (master) $ cd > ../../.. > 4.2.3. ~/workspace/metron (master) $ git pull > 4.2.4. ~/workspace/metron (master) $ mvn clean install -DskipTests -T C4 > 4.2.5. ~/workspace/metron (master) $ cd metron-deployment/development/centos6 > 4.2.4. ~/workspace/metron/metron-deployment/development/centos6 (master) $ > vagrant up > > 4.3. terminal dump > ................................................... > ..--->>>redacted terminal dump for brevity<<<--- .. > ................................................... > [INFO] --- maven-install-plugin:2.4:install (default-install) @ metron-rest > --- > [INFO] Installing > /Users/scottcote/workspace/metron/metron-interface/metron-rest/target/metron-rest-0.6.1.jar > to > /Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1.jar > [INFO] Installing > /Users/scottcote/workspace/metron/metron-interface/metron-rest/pom.xml to > /Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1.pom > [INFO] Installing > /Users/scottcote/workspace/metron/metron-interface/metron-rest/target/metron-rest-0.6.1-archive.tar.gz > to > /Users/scottcote/.m2/repository/org/apache/metron/metron-rest/0.6.1/metron-rest-0.6.1-archive.tar.gz > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] Metron ............................................. SUCCESS [ 14.439 > s] > [INFO] metron-stellar ..................................... SUCCESS [ 1.317 > s] > [INFO] stellar-common ..................................... SUCCESS [ 39.920 > s] > [INFO] metron-analytics ................................... SUCCESS [ 1.221 > s] > [INFO] metron-maas-common ................................. SUCCESS [ 1.142 > s] > [INFO] metron-platform .................................... SUCCESS [ 1.666 > s] > [INFO] metron-zookeeper ................................... SUCCESS [ 5.004 > s] > [INFO] metron-test-utilities .............................. SUCCESS [ 2.454 > s] > [INFO] metron-integration-test ............................ SUCCESS [ 20.909 > s] > [INFO] metron-maas-service ................................ SUCCESS [ 20.476 > s] > [INFO] metron-common ...................................... SUCCESS [ 34.899 > s] > [INFO] metron-statistics .................................. SUCCESS [ 33.713 > s] > [INFO] metron-writer ...................................... SUCCESS [ 32.067 > s] > [INFO] metron-storm-kafka-override ........................ SUCCESS [ 11.703 > s] > [INFO] metron-storm-kafka ................................. SUCCESS [ 2.069 > s] > [INFO] metron-hbase ....................................... SUCCESS [ 3.316 > s] > [INFO] metron-profiler-common ............................. SUCCESS [ 2.298 > s] > [INFO] metron-profiler-client ............................. SUCCESS [ 31.861 > s] > [INFO] metron-profiler-storm .............................. SUCCESS [01:24 > min] > [INFO] metron-profiler-spark .............................. SUCCESS [01:26 > min] > [INFO] metron-profiler-repl ............................... SUCCESS [ 22.633 > s] > [INFO] metron-hbase-client ................................ SUCCESS [ 23.168 > s] > [INFO] metron-enrichment .................................. SUCCESS [01:27 > min] > [INFO] metron-indexing .................................... SUCCESS [ 23.352 > s] > [INFO] metron-solr ........................................ SUCCESS [01:06 > min] > [INFO] metron-job ......................................... SUCCESS [ 1.818 > s] > [INFO] metron-pcap ........................................ SUCCESS [ 4.094 > s] > [INFO] metron-parsers ..................................... SUCCESS [01:14 > min] > [INFO] metron-pcap-backend ................................ SUCCESS [ 54.775 > s] > [INFO] metron-data-management ............................. SUCCESS [01:13 > min] > [INFO] metron-management .................................. SUCCESS [ 12.919 > s] > [INFO] elasticsearch-shaded ............................... SUCCESS [ 25.123 > s] > [INFO] metron-elasticsearch ............................... SUCCESS [01:22 > min] > [INFO] metron-deployment .................................. SUCCESS [ 1.265 > s] > [INFO] Elasticsearch Ambari Management Pack ............... SUCCESS [ 2.529 > s] > [INFO] Metron Ambari Management Pack ...................... SUCCESS [ 3.869 > s] > [INFO] metron-contrib ..................................... SUCCESS [ 1.151 > s] > [INFO] metron-docker ...................................... SUCCESS [ 4.692 > s] > [INFO] metron-performance ................................. SUCCESS [ 4.457 > s] > [INFO] metron-interface ................................... SUCCESS [ 9.772 > s] > [INFO] metron-config ...................................... SUCCESS [05:00 > min] > [INFO] metron-alerts ...................................... SUCCESS [03:48 > min] > [INFO] metron-rest-client ................................. SUCCESS [ 1.148 > s] > [INFO] metron-rest ........................................ SUCCESS [ 33.988 > s] > [INFO] site-book .......................................... SUCCESS [ 1.929 > s] > [INFO] 3rd party Functions (just for tests) ............... SUCCESS [ 0.583 > s] > [INFO] stellar-zeppelin ................................... SUCCESS [ 2.079 > s] > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 05:46 min (Wall Clock) > [INFO] Finished at: 2018-10-23T09:22:06-05:00 > [INFO] Final Memory: 220M/2041M > [INFO] > ------------------------------------------------------------------------ > ~/workspace/metron (master) $ cd metron-deployment/development/centos6/ > ~/workspace/metron/metron-deployment/development/centos6 (master) $ vagrant up > Running with ansible-skip-tags: ["sensors"] > Bringing machine 'node1' up with 'virtualbox' provider... > ==> node1: Importing base box 'centos/6'... > ==> node1: Matching MAC address for NAT networking... > ==> node1: Checking if box 'centos/6' is up to date... > ==> node1: Setting the name of the VM: centos6_node1_1540304760857_3427 > ==> node1: Clearing any previously set network interfaces... > ==> node1: Preparing network interfaces based on configuration... > node1: Adapter 1: nat > node1: Adapter 2: hostonly > ==> node1: Forwarding ports... > node1: 22 (guest) => 2222 (host) (adapter 1) > ==> node1: Running 'pre-boot' VM customizations... > ==> node1: Booting VM... > ==> node1: Waiting for machine to boot. This may take a few minutes... > node1: SSH address: 127.0.0.1:2222 <http://127.0.0.1:2222/> > node1: SSH username: vagrant > node1: SSH auth method: private key > node1: > node1: Vagrant insecure key detected. Vagrant will automatically replace > node1: this with a newly generated keypair for better security. > node1: > node1: Inserting generated public key within guest... > node1: Removing insecure key from the guest if it's present... > node1: Key inserted! Disconnecting and reconnecting using new SSH key... > ==> node1: Machine booted and ready! > ==> node1: Checking for guest additions in VM... > node1: No guest additions were detected on the base box for this VM! Guest > node1: additions are required for forwarded ports, shared folders, host > only > node1: networking, and more. If SSH fails on this machine, please install > node1: the guest additions and repackage the box to continue. > node1: > node1: This is not an error message; everything may continue to work > properly, > node1: in which case you may ignore this message. > ==> node1: Setting hostname... > ==> node1: Configuring and enabling network interfaces... > node1: SSH address: 127.0.0.1:2222 <http://127.0.0.1:2222/> > node1: SSH username: vagrant > node1: SSH auth method: private key > ==> node1: Rsyncing folder: > /Users/scottcote/workspace/metron/metron-deployment/development/centos6/ => > /vagrant > ==> node1: [vagrant-hostmanager:guests] Updating hosts file on active guest > virtual machines... > ==> node1: [vagrant-hostmanager:host] Updating hosts file on your workstation > (password may be required)... > Password: > ==> node1: Running provisioner: ansible... > Vagrant has automatically selected the compatibility mode '2.0' > according to the Ansible version installed (2.2.2.0). > Alternatively, the compatibility mode can be specified in your Vagrantfile: > https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode > > <https://www.vagrantup.com/docs/provisioning/ansible_common.html#compatibility_mode> > node1: Running ansible-playbook... > PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false > ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o > IdentityFile=/Users/scottcote/workspace/metron/metron-deployment/development/centos6/.vagrant/machines/node1/virtualbox/private_key > -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook > --connection=ssh --timeout=30 --extra-vars=ansible_user\=\'vagrant\' > --limit="node1" --inventory-file=ansible/inventory --become -vvv > --skip-tags=sensors ansible/playbook.yml > Using > /Users/scottcote/workspace/metron/metron-deployment/development/centos6/ansible.cfg > as config file > ERROR! no action detected in task. This often indicates a misspelled module > name, or incorrect module path. > The error appears to have been in > '/Users/scottcote/workspace/metron/metron-deployment/ansible/roles/enable-swap/tasks/main.yml': > line 29, column 3, but may > be elsewhere in the file depending on the exact syntax problem. > The offending line appears to be: > - include_tasks: enable-swap.yml > ^ here > The error appears to have been in > '/Users/scottcote/workspace/metron/metron-deployment/ansible/roles/enable-swap/tasks/main.yml': > line 29, column 3, but may > be elsewhere in the file depending on the exact syntax problem. > The offending line appears to be: > - include_tasks: enable-swap.yml > ^ here > Ansible failed to complete successfully. Any error output should be > visible above. Please fix these errors and try again. > > 5. enable solr and disable elastic > > go to `ansible/inventory/group_vars/all` > > and set > `# search > install_elasticsearch: False > install_solr: True > solr_collection_name: Metron > solr_number_shards: 1 > solr_replication_factor: 1 > elasticsearch_transport_port: 9300 > ## The elasticsearch_network_interface must be in the form specified for > Elasticsearch, with leading and trailing underscores. > elasticsearch_network_interface: _eth1_ > elasticsearch_web_port: 9200 > ansible/inventory/group_vars/all > ` > > 6 make install verbose > > mod this file > ~/devprojects/metron/metron-deployment/development/centos6/Vagrantfile > > er, metron/metron-deployment/development/centos6/Vagrantfile > under config.vm.provision add this > ansible.verbose = "vvv" > > > 7. ansible correction based on feedback that I'm using head of master and its > not in synch with the specified version of ansible on the page > > > `pip install ansible==2.5.0` > > then do `vagrant up` >
