Hi Carl, Taking a look over the diagnostics, it looks like you don’t have cassandra installed. I think you’ve been slightly caught out by our package names; the clearwater-cassandra package is actually a wrapper around cassandra itself that provides a number of tweaks to the base cassandra installation for it to work with clearwater, but doesn’t provide the actual cassandra binaries. To get your installation up and running, you will need to source a build of cassandra compatible with your ARM VMs; we don’t have anything available in the repo you’ve been using.
Once that is set up, the nodes should sort themselves out and have everything startup cleanly. If not, we may need to do a bit more work setting up the clusters to get them in a healthy state, and making sure that the clearwater-cassandra package is interacting with the different cassandra version as we expect it to. Let us know if you manage to get it up and running Cheers, Adam From: Clearwater [mailto:[email protected]] On Behalf Of Persson, Carl J [CTO] Sent: 12 October 2017 19:06 To: [email protected] Subject: Re: [Project Clearwater] Clearwater on ARM Ubuntu 16.04 VMs Thanks Adam - I really appreciate the help! Here is the info that you requested. My first reply apparently got hung up because it was too big and not approved for posting so I am breaking my response into pieces. The dpkg -l was too long. I do suspect that Cassandra is not installed properly, but if so, not sure where to get the correct package. These are the instructions I was provided for the Homestead configuration. If on a Homestead node, install Java and Cassandra First, copy the three files you downloaded ealier (python-support_1.0.15_all.deb, jdk9-server-release-1605.tar.xz and cassandra_3.5_cavium.deb) to the current directory, then run: sudo apt-get install default-jre openjdk-8-jre python-pip jemalloc && sudo dpkg -i python-support_1.0.15_all.deb && tar xJf jdk9-server-release-1605.tar.xz && sudo mkdir /usr/jdk && sudo mv jdk9-server-release-1605 /usr/jdk/ && sudo chown -R root.root /usr/jdk && sudo chmod -R 755 /usr/jdk/ && sudo cp -R /usr/jdk/jdk9-server-release-1605 /usr/lib/jvm/java9-openjdk-1605 && sudo dpkg -i cassandra_3.5_cavium.deb && sudo rm -f /usr/lib/pymodules/python2.7/cqlshlib/copyutil.so && sudo pip install cassandra-driver sudo DEBIAN_FRONTEND=noninteractive apt-get install homestead homestead-prov clearwater-prov-tools -y && sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management -y && sudo systemctl daemon-reload For the Cassandra part, I downloaded and installed clearwater-cassandra from : http://repo.cw-ngv.com/~aarch64/repo/binary/ Also, the jemalloc install failed because it couldn’t find it. I searched quite a bit on the internet and found and installed: sudo apt install libjemalloc-dev I don’t know if that is the correct package. There is no /etc/cassandra directory, I also noticed that there is no /var/log/cassandra directory either. So even though it appears that Cassandra is running, things don’t appear to look correct. Here is the monit summary: [homestead]sprint@homestead:/etc$ sudo monit summary The Monit daemon 5.8.1 uptime: 5d 1h 44m Process 'ntp_process' Running System 'node-homestead' Running Process 'nginx_process' Running Program 'nginx_uptime' Waiting Program 'monit_uptime' Waiting Process 'homestead_process' Execution failed Program 'homestead_uptime' Not monitored Program 'poll_homestead' Not monitored Process 'homestead-prov_process' Running Program 'poll_homestead-prov' Status ok Process 'clearwater_queue_manager_process' Running Program 'clearwater_queue_manager_uptime' Status ok Process 'etcd_process' Running Program 'etcd_uptime' Status ok Program 'poll_etcd_cluster' Status ok Program 'poll_etcd' Status ok Process 'clearwater_diags_monitor_process' Running Process 'clearwater_config_manager' Running Process 'clearwater_cluster_manager' Running Process 'cassandra_process' Execution failed Program 'cassandra_uptime' Initializing Program 'poll_cassandra’ Initializing Local and shared logs: [homestead]sprint@homestead:/etc/clearwater$ cat local_config local_ip=192.168.122.41 public_ip=192.168.122.41 public_hostname=192.168.122.41 etcd_cluster=192.168.122.49 # use an IP address of a node whose cluster you intend to join [homestead]sprint@homestead:/etc/clearwater$ cat shared_config # Deployment definitions #home_domain=cavium.obu-enfield.test sprout_hostname=192.168.122.49 hs_hostname=192.168.122.41:8888 hs_provisioning_hostname=192.168.122.41:8889 #ralf_hostname=ralf.cavium.obu-enfield.test:10888 #xdms_hostname=homer.cavium.obu-enfield.test:7888 turn_workaround=secret On Oct 6, 2017, at 9:47 AM, Adam Lindley <[email protected]<mailto:[email protected]>> wrote: Hi Carl, Interesting problem. We don’t have many people looking at running clearwater up on ARM, so it’ll be good to work out what’s going on here. As a first step, could you provide the following pieces of information: • The steps you took when installing the Java and Cassandra packages, and which packages you are working with (as we have seen some issues in the past with these) • The Cassandra.yaml file, under /etc/cassandra/ • The output of `sudo monit summary` on the node, so we can see what processes are running/attempting to run • The contents of the local_config and shared_config files on the Homestead node, under /etc/clearwater/ • The list of packages and versions installed on the system (something like dpkg -l , or dpkg-query -W) Potentially there could be a problem stopping the cluster-manager from starting up correctly, which would stop it generating the config necessary for Cassandra to start up properly (the Cassandra.yaml file is automatically re-written using information taken from the etcd cluster). The fact that there is no log under /var/log/cassandra/ suggests that the service is not starting up at all. This might be caused by an unhealthy etcd cluster: If the output of `sudo monit summary` suggests that the etcd process is not healthy, we should start by looking into that, with logs present in /var/log/clearwater-etcd/; if the process does look healthy we can also use `clearwater-etcdctl cluster-health` and `clearwater-etcdctl member-list` to verify that the cluster is both healthy and contains the nodes that we expect. If etcd is unstable, you can try running it manually by taking it out of Monit’s control, by running `sudo monit unmonitor –g etcd`, and then running `sudo service clearwater-etcd start` Cheers, Adam From: Clearwater [mailto:[email protected]] On Behalf Of Persson, Carl J [CTO] Sent: 05 October 2017 17:06 To: [email protected]<mailto:[email protected]> Subject: [Project Clearwater] Clearwater on ARM Ubuntu 16.04 VMs I am trying to install Clearwater on Ubuntu 16.04 ARM VMs, and I am running into problems with the Homestead node. I installed clearwater-cassandra from : http://repo.cw-ngv.com/~aarch64/repo/binary/<https://na01.safelinks.protection.outlook.com/?url=http:%2F%2Frepo.cw-ngv.com%2F~aarch64%2Frepo%2Fbinary%2F&data=02%7C01%7Ccarl.j.persson%40sprint.com%7C351a481b9b884cf735e908d50cc96484%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636428981472554313&sdata=TrGJ8nporbVdIeKkh%2FakxXVdp7QVcV2KaiQQP%2FmMwaI%3D&reserved=0> It appears to be running: [homestead]sprint@homestead:/etc$ sudo service cassandra status ● cassandra.service - LSB: distributed storage system for structured data Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled) Active: active (exited) since Thu 2017-10-05 11:57:31 EDT; 1min 9s ago Docs: man:systemd-sysv-generator(8) Process: 26433 ExecStart=/etc/init.d/cassandra start (code=exited, status=0/SUCCESS) Tasks: 0 Memory: 0B CPU: 0 Oct 05 11:57:31 homestead systemd[1]: Starting LSB: distributed storage system for structured data... Oct 05 11:57:31 homestead systemd[1]: Started LSB: distributed storage system for structured data. But the monit.log file says the the Cassandra process isn’t running and can’t start. I’m not sure what the problem is. [EDT Oct 5 11:57:03] error : 'cassandra_process' process is not running [EDT Oct 5 11:57:03] info : 'cassandra_process' trying to restart [EDT Oct 5 11:57:03] info : 'cassandra_process' restart: /bin/bash [EDT Oct 5 11:57:33] error : 'cassandra_process' failed to start [EDT Oct 5 11:57:33] info : 'cassandra_process' start: /bin/bash [EDT Oct 5 11:58:03] error : 'cassandra_process' failed to start [EDT Oct 5 11:58:13] error : 'homestead_process' process is not running [EDT Oct 5 11:58:13] info : 'homestead_process' trying to restart [EDT Oct 5 11:58:13] info : 'homestead_process' restart: /bin/bash I also, do not see a cassandra log file in /var/log or anywhere that I can find. Has anyone had success installing Clearwater on the ARM platform that could provide some guidance: Thanks, Carl ________________________________ This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message. _______________________________________________ Clearwater mailing list [email protected]<mailto:[email protected]> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.projectclearwater.org%2Fmailman%2Flistinfo%2Fclearwater_lists.projectclearwater.org&data=02%7C01%7Ccarl.j.persson%40sprint.com%7C351a481b9b884cf735e908d50cc96484%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636428981472554313&sdata=yA0oN%2BOBlrSqRymR1FSY61ihjDSUzMiBoyBhgXHcuIc%3D&reserved=0
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
