Thao, Thanks for your response.
Yes, if you've successfully created a user through Ellis, that means that Homestead-prov and its Cassandra and also Homer and its Cassandra must all be running. Yes, please let me know how you get on. Thanks, Matt From: Thao HT Nguyen [mailto:[email protected]] Sent: 01 July 2015 18:42 To: Matt Williams Cc: [email protected] Subject: Re: [Clearwater] clearwater-docker: issues with Homestead: port 8888 not available (and other issues) Hi Matt, Thanks again for your reply. For the SCTP support: I said I was running from my MacOS, but it's actually running on top of Boot2Docker VM which is using a Tiny Core Linux distribution. That might be the reason why I miss the SCTP kernel support. I will try another way and perhaps boot a Vagrant machine with Ubuntu w/ Docker on it. For the ulimit message: I mentioned I couldn't start Cassandra on Homer and tried to start it manually. I noticed the ulimit error at that time and that was before https://github.com/Metaswitch/clearwater-docker/pull/9 update. I wasn't aware of the fact that the daemon would go through despite of the error. In the end, if I managed to create a user, this means that "Homer/Cassandra" is running correctly, isn't it? I will let you know of the outcome with Ubuntu on top of my Vagrant machine. Thanks again. Thao From: Matt Williams <[email protected]<mailto:[email protected]>> Date: Wednesday, July 1, 2015 at 2:13 AM To: Thao HT Nguyen <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: [Clearwater] clearwater-docker: issues with Homestead: port 8888 not available (and other issues) Thao, Thanks for your email. I'm glad to hear the Ellis issue is now fixed. The Homestead issue seems to be that your kernel doesn't support SCTP. From homestead.log: 30-06-2015 15:00:05.306 UTC Status diameterstack.cpp:475: Starting Diameter stack 30-06-2015 15:00:05.316 UTC Error freeDiameter: ERROR: in '(*sock = socket(family, SOCK_STREAM, IPPROTO_SCTP))' : Protocol not supported 30-06-2015 15:00:05.317 UTC Error freeDiameter: ERROR: in '(fd_sctp_create_bind_server( &cnx->cc_socket, cnx->cc_family, ep_list, port ))' : Protocol not supported 30-06-2015 15:00:05.317 UTC Error freeDiameter: ERROR: in '(s->conn = fd_cnx_serv_sctp(fd_g_config->cnf_port, empty_conf_ep ? ((void *)0) : &fd_g_config->cnf_endpoints))' : Protocol not supported 30-06-2015 15:00:05.317 UTC Error freeDiameter: ERROR: in '((fd_servers_start()))' : Protocol not supported 30-06-2015 15:00:05.317 UTC Error main.cpp:732: Failed to initialize Diameter stack - function fd_core_start, rc 93 freeDiameter (the Diameter stack which Homestead uses for the Cx interface to the HSS) expects SCTP kernel support. Can you enable SCTP in your kernel? (I'm afraid I don't know much about running Docker on top of MacOS.) If it's not possible to enable SCTP under MacOS, you could try modifying /var/lib/homestead/homestead.conf to add a line saying "No_SCTP;" and then try re-running Homestead. (Note that this file is overwritten on boot - see /usr/share/clearwater/bin/generic_create_diameterconf for the file that generates this.) On the ulimit error you saw, this should be benign - they are setting a couple of scalability options (which you're unlikely to hit at the scale you'd run under Docker, as Docker encourages many smaller containers as opposed to fewer large VMs) and a core file generation option. The start-up scripts should run straight through the error and start the daemon anyway (although in this case it wasn't starting due to lack of SCTP support). You mentioned that Homer wasn't running either - can you share diagnostics for that? Thanks, Matt From: Thao HT Nguyen [mailto:[email protected]] Sent: 30 June 2015 21:11 To: Matt Williams Cc: [email protected]<mailto:[email protected]> Subject: Re: [Clearwater] clearwater-docker: issues with Homestead: port 8888 not available (and other issues) Hello Matt, Please read below (inline) Thao From: Matt Williams <[email protected]<mailto:[email protected]>> Date: Tuesday, June 30, 2015 at 4:34 AM To: Thao HT Nguyen <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: [Clearwater] clearwater-docker: issues with Homestead: port 8888 not available (and other issues) Thao, Thanks for raising these. I haven't seen the Homestead issue you describe. * Are you able to share the install/run output logs? [Thao] Sure, they are attached in this email. * Also, what do you see if you run "/etc/init.d/homestead run" from the homestead node's command-line? [Thao] This is one of the other issues I didn't mention yet. I could not run it manually on "homer" nor "homestead". If you look at build_homestead.txt, you will see an ulimit error. Run from my box: root@a43376332100:~# /etc/init.d/homestead run /etc/init.d/homestead: 84: ulimit: error setting limit (Invalid argument) Logs from homestead during build: Step 4 : RUN /etc/init.d/cassandra start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes homestead homestead-prov ---> Running in 6aacca1aa389 /etc/init.d/cassandra: 84: ulimit: error setting limit (Operation not permitted) * Which version of clearwater-docker are you running? There are three issues it might be related to, but they've been fixed - which version of clearwater-docker are you running against? [Thao] I always pull the latest from the master branch every time I rebuild everything (I made sure that I removed all the images as well) bash-3.2$ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working directory clean bash-3.2$ git branch -av * master fd2727e Merge pull request #13 from Metaswitch/graeme_proofing remotes/origin/HEAD -> origin/master remotes/origin/master fd2727e Merge pull request #13 from Metaswitch/graeme_proofing remotes/origin/stable 88a23c0 Merge pull request #11 from Metaswitch/issue10 * https://github.com/Metaswitch/clearwater-docker/pull/9 covers an upgrade to work correctly with Cassandra 2.0, but this went in about 3 weeks ago. [Thao] I know, I was the one who raised that issue ;) * https://github.com/Metaswitch/clearwater-docker/issues/10 covers the fact that we used not to enforce a strict ordering of start-up process. I've never seen this affect Cassandra or Homestead, though. * If you were running from the "latest" repo rather than the standard "stable" repo (although I don't think you are), you might have hit the issue that is addressed by https://github.com/Metaswitch/homestead/pull/234 - basically, we weren't specifying a dependency that is normally satisfied by standard Ubuntu VMs, but not by Docker's stripped-down base image. This is now fixed in the new "latest" repo. Also, I noticed that "monit status" reports output on your box - did you start monit manually? (Monit doesn't run well under Docker, so we use supervisord instead.) [Thao] Ah oK! Yeah, for some reason, monit never started. So I did it manually. I think the Ellis issue is https://github.com/Metaswitch/clearwater-docker/issues/12, which was fixed under https://github.com/Metaswitch/clearwater-infrastructure/pull/202. This is broken in the "stable" repo, but is fixed in the "latest" repo. To switch to the "latest" repo, edit https://github.com/Metaswitch/clearwater-docker/blob/master/base/Dockerfile#L19 to replace "http://repo.cw-ngv.com/stable"; with "http://repo.cw-ngv.com/latest";. You'll then need to re-build all the images (probably by deleting the images using "docker rmi" and then re-running the build process). [Thao] Ok, I made the changes in the "base" docker file and it fixed the domain issue. One other question: which platform are you running Docker on? We've seen problems on some CoreOS systems relating to a bug in OverlayFS that breaks UNIX domain sockets - see https://github.com/coreos/bugs/issues/311 for details. [Thao] I am using MacOS to run Docker. I hope that helps - please let me know how you get on! We only recently made our Docker support public, and don't yet have great regression tests for it - we're hoping to flesh this out in the near future. Thanks, Matt From:[email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Thao HT Nguyen Sent: 30 June 2015 00:31 To: [email protected]<mailto:[email protected]> Subject: [Clearwater] clearwater-docker: issues with Homestead: port 8888 not available (and other issues) Hello, I would like to deploy and run the clearwater-docker by applying the docker commands suggested in the README.md on https://github.com/Metaswitch/clearwater-docker/tree/master/. I had several issues, so let's start with the first ones (I will email other issues once I pass through these ones). _________________________________________________________________________________________________________ ISSUES WITH HOMESTEAD: Cassandra (port 8888) not running. I found out that homestead port 8888 was not available. Here is the list of processes captured on homestead: root 1 0 0 23:01 ? 00:00:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf root 7 1 0 23:01 ? 00:00:00 /usr/sbin/sshd -D root 147 1 0 23:01 ? 00:00:00 /bin/sh /etc/init.d/homestead-prov run root 148 1 0 23:01 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon off; root 151 147 0 23:01 ? 00:00:01 /usr/share/clearwater/homestead/env/bin/python -m metaswitch.crest.main --worker-processes 1 www-data 152 148 0 23:01 ? 00:00:00 nginx: worker process www-data 153 148 0 23:01 ? 00:00:00 nginx: worker process www-data 154 148 0 23:01 ? 00:00:00 nginx: worker process www-data 155 148 0 23:01 ? 00:00:00 nginx: worker process root 186 1 4 23:01 ? 00:00:54 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1001M -Xmx1001M -Xmn250M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX root 970 7 0 23:19 ? 00:00:00 sshd: root@pts/0 root 981 970 0 23:19 pts/0 00:00:00 -bash root 1003 1 0 23:19 ? 00:00:00 monit root 1030 1 0 23:19 ? 00:00:00 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 103:106 root 1099 1 0 23:19 ? 00:00:00 /bin/bash /usr/share/clearwater/bin/clearwater_diags_monitor root 1104 1099 0 23:19 ? 00:00:00 inotifywait -e create -qq . root 5045 1003 2 23:21 ? 00:00:00 [poll_homestead-] <defunct> cassand+ 5188 1 99 23:21 ? 00:00:04 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1001M -Xmx1001M -Xmn250M -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX root 5383 981 0 23:21 pts/0 00:00:00 ps -ef Here is part of the "monit status": Process 'homestead_process' status Does not exist monitoring status Monitored data collected Mon, 29 Jun 2015 23:20:14 Process 'cassandra_process' status Does not exist monitoring status Monitored data collected Mon, 29 Jun 2015 23:20:15 _________________________________________________________________________________________________________ ISSUES WITH ELLIS (or maybe the issues comes from the "base" build?) - Redeployed from scractch last week - The provisioning portal (ellis) doesn't display the domain name anymore. [cid:[email protected]] I found during the "base" build that following error: Step 18 : RUN /etc/init.d/clearwater-auto-config-docker restart ---> Running in 38123f560712 /etc/init.d/clearwater-auto-config-docker: 82: /etc/init.d/clearwater-auto-config-docker: home_domain: not found ---> c43db9b6e245 Removing intermediate container 38123f560712 Step 19 : RUN /etc/init.d/clearwater-infrastructure restart ---> Running in 12a56bd101d0 * Restarting clearwater-infrastructure clearwater-infrastructure Configuring monit for only localhost access ...done. ________________________________ Please let me know how I can fix those issues. Thanks in advance Thao
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/listinfo/clearwater
