Hi Ellie, 1. LOG_LEVEL=logging.DEBUG and yes I run it.
2. This is my shared config, I run the two commands to restart infrastructure and stop ellis and still it generates subscribers with the domain 'cw-ngv.com' # Deployment definitions home_domain=example.com sprout_hostname=sprout.example.com hs_hostname=hs.example.com:8888 hs_provisioning_hostname=hs.example.com:8889 ralf_hostname=ralf.example.com:10888 xdms_hostname=homer.example.com:7888 # Email server configuration smtp_smarthost=localhost smtp_username=username smtp_password=password [email protected] # Keys signup_key=secret turn_workaround=secret ellis_api_key=secret ellis_cookie_key=secret 3. Regarding Bono I did what you suggested but it says bono_process does not exists, no process is listening on port 5060 or 5062 whaen running 'netstat -na' and in /var/log/syslog there is the following error - Oct 5 11:59:29 bono-0 bono[20299]: 2006 - Description: Fatal - Must enable P-CSCF, S-CSCF or I-CSCF in /etc/clearwater/config. @@Cause: Neither a P-CSCF, a S-CSCF nor an I-CSCF was configured in /etc/clearwater/config. @@Effect: The application will exit and restart until the problem is fixed. @@Action: The P-CSCF is configured by setting the pcscf=<port> option. The S-CSCF is configured by setting the scscf=<port> option. The I-CSCF is configured by setting the icscf=<port> option. ~ I saw something similar to that here in sprout - Error main.cpp:887: Must enable P-CSCF, S-CSCF or I-CSCF in https://github.com/Metaswitch/sprout/issues/450 I copied the shared_config to bono from ellis since there was no shared_config created in bono. What is urgent for me is to be able to provision subscribers (I thing that part is done except for the cw-ngv.coma domain and I'd like to have example.com) and be able to connect tow Jitsi clients and demonstrate a video call. Thanks, Shay From: Eleanor Merry [mailto:[email protected]] Sent: Monday, October 5, 2015 12:49 PM To: Shay Naeh Cc: [email protected] Subject: RE: Deploying Clearwater with Haet Hi Shay, I've put answers to your points below: 1. Just to check, you did run 'sudo service clearwater-infrastructure restart' then 'sudo service ellis stop' after you changed the local_settings.py file? What's the value of LOG_LEVEL in '/usr/share/clearwater/ellis/env/lib/python2.7/site-packages/ellis-0.1-py2.7.egg/metaswitch/ellis/local_settings.py'? 2. 'cw-ngv.com' is the default value Ellis uses for the domain. It's overridden by the value of home_domain in the /etc/clearwater/shared_config file - to trigger the overriding you do need to run the 'sudo service clearwater-infrastructure restart' then 'sudo service ellis stop' commands (same as for turning on debug logging). 3. Did you run the '/usr/share/clearwater/clearwater-config-manager/scripts/apply_shared_config' script on Bono? You need to do this for Bono to pick up the changes (this runs 'sudo service clearwater-infrastructure restart' then 'sudo service bono stop' under the covers, so it's the same steps that are necessary for Ellis). Ellie From: Shay Naeh [mailto:[email protected]] Sent: 03 October 2015 23:39 To: Eleanor Merry Subject: RE: Deploying Clearwater with Haet Hi Ellie, 1. I checked the flag and its value in local_settings.py is "LOG_LEVEL = logging.DEBUG" don't understand why we don't get debug info 2. You were right regarding the timeouts I fixed the connectivity issue and I am able to provision new subscribes (after truncating the DBs). It provision them with the domain [email protected]<mailto:[email protected]> cw-ngv.com, why not example.com? 3. I have issues with bono. Shared_config was missing in bono and I copied it from ellis. When running 'sudo monit status' it says bono_process does not exist and in the /var/log/syslog I find the following error: Oct 3 06:45:02 bono-0 bono[13215]: 2006 - Description: Fatal - Must enable P-CSCF, S-CSCF or I-CSCF in /etc/clearwater/config. @@Cause: Neither a P-CSCF, a S-CSCF nor an I-CSCF was configured in /etc/clearwater/config. @@Effect: The application will exit and restart until the problem is fixed. @@Action: The P-CSCF is configured by setting the pcscf=<port> option. The S-CSCF is configured by setting the scscf=<port> option. The I-CSCF is configured by setting the icscf=<port> option. Oct 3 06:45:12 bono-0 bono[13239]: 2005 - Description: Application started. @@Cause: The application is starting. @@Effect: Normal. @@Action: None. Thanks, Shay From: Eleanor Merry [mailto:[email protected]] Sent: Friday, October 2, 2015 7:47 PM To: Shay Naeh Subject: RE: Deploying Clearwater with Haet Hi Shay, There's still no debug logs. Can you confirm that you edited the local settings file, restarted clearwater-infrastructure and restarted Ellis? In the logs though there's a lot of timeouts to hs.example.com:8889 and homer.example.com:7888. Can you confirm that these are the correct hostnames for your Homestead and Homer nodes, and that the Ellis node can successfully talk to the Homestead node over port 8889, and to the Homer node at port 7888. Ellie From: Shay Naeh [mailto:[email protected]] Sent: 02 October 2015 17:37 To: Eleanor Merry Subject: RE: Deploying Clearwater with Haet Logs tar file attached. Thanks, Shay From: Eleanor Merry [mailto:[email protected]] Sent: Friday, October 2, 2015 6:35 PM To: Shay Naeh; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Ok, can you please send me the debug logs from Ellis? This will help narrow down whether the problem is with Ellis, Homer or Homestead. To turn on debug logging for Ellis, write LOG_LEVEL = logging.DEBUG to the local_settings.py file (at /usr/share/clearwater/ellis/src/metaswitch/ellis/local_settings.py). Then restart clearwater-infrastructure (sudo service clearwater-infrastructure restart), and restart Ellis (sudo service ellis stop - it will be restarted by monit). Ellie From: Shay Naeh [mailto:[email protected]] Sent: 02 October 2015 16:25 To: Eleanor Merry; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet I've deleted all the numbers and when login to ellis it tells me you have no numbers, create one. When trying to create I get the same error failed to update the server. And yes I have shared_config on all nodes Thanks, Shay From: Eleanor Merry [mailto:[email protected]] Sent: Friday, October 2, 2015 6:07 PM To: Shay Naeh; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Hi, It looks like there is a corrupted number in the Ellis/Homestead databases. Can you please remove the numbers? You can do this by running: To delete all numbers on Ellis, run the following: sudo mysql USE ellis UPDATE numbers SET owner_id=NULL; To delete all numbers on Homestead, run the following: cqlsh USE homestead_provisioning; TRUNCATE public; TRUNCATE private; TRUNCATE service_profiles; TRUNCATE implicit_registration_sets; USE homestead_cache; TRUNCATE impi; TRUNCATE impu; TRUNCATE impi_mapping; To delete all numbers on Homer, run the following: cqlsh USE homer; TRUNCATE simservs; Can you also please confirm that the shared_configuration file is present on all your nodes? Ellie From: Shay Naeh [mailto:[email protected]] Sent: 02 October 2015 15:27 To: Robert Day; [email protected]<mailto:[email protected]> Cc: Eleanor Merry Subject: RE: Deploying Clearwater with Haet Hi Rob and Ellie, 1. Sudo service homer restart didn't help, but Sudo monit start homer_proccess worked just fine. 2. Regarding ellis I played with it and what helped bringing up the Login screen was after provisioning additional phone numbers (don't understand why it is related). 3. Now I encounter an issue after the login, I get the error 'failed to update the server (see detailed diagnostics in the developer console)". I think I saw an error like that that was related to the Cassandra DB tables but I suspect that some configuration regarding IPs or FQDNS is wrong. In ellis log files I find the following uncaught exception. Thanks, Shay 02-10-2015 14:15:22.080 UTC INFO web.py:1447: 200 GET /ping (0.0.0.0) 17.18ms 02-10-2015 14:16:02.117 UTC INFO web.py:1447: 200 GET /ping (0.0.0.0) 0.37ms 02-10-2015 14:16:42.169 UTC INFO web.py:1447: 200 GET /ping (0.0.0.0) 0.46ms 02-10-2015 14:17:22.213 UTC INFO web.py:1447: 200 GET /ping (0.0.0.0) 0.28ms 02-10-2015 14:17:40.600 UTC INFO web.py:1447: 200 GET /css/bootstrap-responsive.css (0.0.0.0) 2.12ms 02-10-2015 14:17:40.822 UTC INFO web.py:1447: 200 GET /css/style.css (0.0.0.0) 1.49ms 02-10-2015 14:17:40.954 UTC INFO web.py:1447: 200 GET /js/jquery.cookie.js (0.0.0.0) 0.66ms 02-10-2015 14:17:41.133 UTC INFO web.py:1447: 200 GET /js/app.js (0.0.0.0) 5.38ms 02-10-2015 14:17:43.362 UTC WARNING homestead.py:309: Passing SIP password in the clear over http 02-10-2015 14:18:03.366 UTC WARNING homestead.py:309: Passing SIP password in the clear over http 02-10-2015 14:18:23.371 UTC WARNING utils.py:78: Non-OK HTTP response. HTTP 599: Timeout 02-10-2015 14:18:23.371 UTC WARNING numbers.py:195: Failed to update all the backends 02-10-2015 14:18:23.371 UTC WARNING homestead.py:309: Passing SIP password in the clear over http 02-10-2015 14:18:23.372 UTC WARNING utils.py:78: Non-OK HTTP response. HTTP 599: Timeout 02-10-2015 14:18:23.373 UTC ERROR iostream.py:307: Uncaught exception, closing connection. Traceback (most recent call last): File "/usr/share/clearwater/ellis/env/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/iostream.py", line 304, in wrapper callback(*args) File "/usr/share/clearwater/ellis/env/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/httpserver.py", line 243, in _on_headers remote_ip = self.address[0] IndexError: string index out of range 02-10-2015 14:18:23.373 UTC ERROR ioloop.py:435: Exception in callback <tornado.stack_context._StackContextWrapper object at 0x7fc2624aedb8> Traceback (most recent call last): File "/usr/share/clearwater/ellis/env/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/ioloop.py", line 421, in _run_callback callback() File "/usr/share/clearwater/ellis/env/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/iostream.py", line 304, in wrapper callback(*args) File "/usr/share/clearwater/ellis/env/local/lib/python2.7/site-packages/tornado-2.3-py2.7.egg/tornado/httpserver.py", line 243, in _on_headers remote_ip = self.address[0] From: Robert Day [mailto:[email protected]] Sent: Thursday, October 1, 2015 10:41 PM To: Shay Naeh; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Hi Shay, Good news that your Homestead node is now OK! For your Homer node, do the log files in /var/log/homer contain any errors that help explain why it isn't starting? What happens if you run "sudo service homer restart" manually - does that print any errors? On the Ellis node, the /usr/share/clearwater/infrastructure/scripts/create-ellis-nginx-config script should create the Nginx config to redirect it to Ellis. Running 'sudo service clearwater-infrastructure' should have run that script, but what happens if you run /usr/share/clearwater/infrastructure/scripts/create-ellis-nginx-config manually? Does it create a /etc/nginx/sites-available/ellis file? If not, can you send us the contents of /etc/clearwater/config, /etc/clearwater/shared_config and /etc/clearwater/local_config? Thanks, Rob -- Rob Day Software Engineer, Project Clearwater From: Clearwater [mailto:[email protected]] On Behalf Of Shay Naeh Sent: 01 October 2015 19:57 To: [email protected]<mailto:[email protected]> Subject: [Clearwater] FW: Deploying Clearwater with Haet Hi ellie, 1. I had to copy the shared_config from ellis to homestead and now it is ok 2. I had the same problem on homer but after copying the shared_config and restarting Clearwater-infrastructure it tells me when doing 'sudo monit status' thar homer_process failed 3. When I point with the browser to Ellis IP address I get the welcome message of Nginx but no login screen as I am used to get. What application should run on top of Nginx? Thanks, Shay From: Eleanor Merry [mailto:[email protected]] Sent: Thursday, October 1, 2015 7:59 PM To: Shay Naeh; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Hi, Is your etcd cluster healthy? What's the result of running sudo clearwater-etcdctl cluster-health? If these look OK then can you also please try running 'sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config' on the Ellis node, then the apply_shared_config script again on Homestead. If this still doesn't work, you can copy over the shared_config file on Ellis to Homestead, run 'sudo service clearwater infrastructure' and this should unblock you. I'd be interested to dig further into why the apply_shared_config isn't working though. Ellie From: Shay Naeh [mailto:[email protected]] Sent: 01 October 2015 17:48 To: Eleanor Merry Subject: RE: Deploying Clearwater with Haet It doesn't exist on Homestead but exists on Ellis. I did clearwater-infrastructure restart but it didn't help. Can we conduct a short webex, do you want me to send you webex details? Thanks, Shay From: Eleanor Merry [mailto:[email protected]] Sent: Thursday, October 1, 2015 7:46 PM To: Shay Naeh; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Does the /etc/clearwater/shared_config file exist on your Homestead node? Does it exist on the Ellis node? If the shared_config file does exist, can you run 'sudo service clearwater-infrastructure restart'? Ellie From: Shay Naeh [mailto:[email protected]] Sent: 01 October 2015 17:22 To: Eleanor Merry; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Thanks Eleanor, I run the /usr/share/clearwater/clearwater-config-manager/scripts/apply_shared_config But still I get the following errors when running 'sudo monit status' Process 'homestead_process' status Does not exist monitoring status Monitored data collected Thu, 01 Oct 2015 16:13:59 Program 'poll_homestead' status Initializing monitoring status Initializing data collected Thu, 01 Oct 2015 16:10:49 Process 'homestead-prov_process' status Execution failed monitoring status Monitored data collected Thu, 01 Oct 2015 16:13:59 Shay From: Eleanor Merry [mailto:[email protected]] Sent: Thursday, October 1, 2015 12:54 PM To: Shay Naeh; [email protected]<mailto:[email protected]> Subject: RE: Deploying Clearwater with Haet Hi Shay, It doesn't look like /etc/clearwater/shared_config has successfully propagated to the Homestead node. In order to set up the diameter configuration file (used by Homestead) and the local_settings file (used by Homestead-prov) correctly, Homestead/Homestead-prov need information from the /etc/clearwater/shared_config file (e.g. the hs_hostname). If the shared_config file is missing; the install will succeed, but Homestead/Homestead-prov won't be functional until they've got their configuration files set up. Looking at the template, we install the clearwater-management package after we install the Homestead/Homestead-prov packages, but don't then do anything to apply the shared configuration to the node. I've raised an issue (https://github.com/Metaswitch/clearwater-heat/issues/15) to fix this; in the meantime can you please run "sudo /usr/share/clearwater/clearwater-config-manager/scripts/apply_shared_config" on all your nodes? Ellie From: Clearwater [mailto:[email protected]] On Behalf Of Shay Naeh Sent: 29 September 2015 16:10 To: [email protected]<mailto:[email protected]> Subject: [Clearwater] Deploying Clearwater with Haet Hello, I am trying deploying Clearwater on my OpenStack using the Heat templates as defined in here https://github.com/Metaswitch/clearwater-heat I am running into issues with Homestead where I get (see below): 1. Why LOCAL_IP=MUST BE CONFIGURED is not configured? 2. Regarding the Diameter stack I saw an open issue https://github.com/Metaswitch/homestead/issues/73 but when deleting the certificates and starting the Clearwater-infrastructure it fails again on the same problem. Thanks, Shay 4007 - Description: Fatal - Failed to initialize Diameter stack in function fd_core_parseconf with error 2. @@Cause: The Diameter interface could not be initialized or encountered an error while running. @@Effect: The application will exit and restart until the problem is fixed. @@Action: (1). Check the configuration for the Diameter destination hosts. (2). Check the connectivity to the Diameter host using Wireshark. Sep 29 06:39:55 homestead-0 monit: Traceback (most recent call last): Sep 29 06:39:55 homestead-0 monit: File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main Sep 29 06:39:55 homestead-0 monit: "__main__", fname, loader, pkg_name) Sep 29 06:39:55 homestead-0 monit: File "/usr/lib/python2.7/runpy.py", line 72, in _run_code Sep 29 06:39:55 homestead-0 monit: exec code in run_globals Sep 29 06:39:55 homestead-0 monit: File "/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py", line 50, in <module> Sep 29 06:39:55 homestead-0 monit: from metaswitch.crest import api Sep 29 06:39:55 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/api/__init__.py", line 35, in <module> Sep 29 06:39:55 homestead-0 monit: from metaswitch.crest.api import base Sep 29 06:39:55 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/api/base.py", line 48, in <module> Sep 29 06:39:55 homestead-0 monit: from metaswitch.crest import settings Sep 29 06:39:55 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/settings.py", line 139, in <module> Sep 29 06:39:55 homestead-0 monit: execfile(_local_settings_file) Sep 29 06:39:55 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/local_settings.py", line 55, in <module> Sep 29 06:39:55 homestead-0 monit: LOCAL_IP = MUST_BE_CONFIGURED Sep 29 06:39:55 homestead-0 monit: NameError: name 'MUST_BE_CONFIGURED' is not defined Sep 29 06:39:55 homestead-0 monit: ...fail! Sep 29 06:40:01 homestead-0 CRON[15030]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d` > /dev/null 2>&1) Sep 29 06:40:22 homestead-0 monit: CMD /etc/init.d/homestead-prov start Sep 29 06:40:26 homestead-0 monit: Traceback (most recent call last): Sep 29 06:40:26 homestead-0 monit: File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main Sep 29 06:40:26 homestead-0 monit: "__main__", fname, loader, pkg_name) Sep 29 06:40:26 homestead-0 monit: File "/usr/lib/python2.7/runpy.py", line 72, in _run_code Sep 29 06:40:26 homestead-0 monit: exec code in run_globals Sep 29 06:40:26 homestead-0 monit: File "/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py", line 50, in <module> Sep 29 06:40:26 homestead-0 monit: from metaswitch.crest import api Sep 29 06:40:26 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/api/__init__.py", line 35, in <module> Sep 29 06:40:26 homestead-0 monit: from metaswitch.crest.api import base Sep 29 06:40:26 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/api/base.py", line 48, in <module> Sep 29 06:40:26 homestead-0 monit: from metaswitch.crest import settings Sep 29 06:40:26 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/settings.py", line 139, in <module> Sep 29 06:40:26 homestead-0 monit: execfile(_local_settings_file) Sep 29 06:40:26 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/local_settings.py", line 55, in <module> Sep 29 06:40:26 homestead-0 monit: LOCAL_IP = MUST_BE_CONFIGURED Sep 29 06:40:26 homestead-0 monit: NameError: name 'MUST_BE_CONFIGURED' is not defined Sep 29 06:41:01 homestead-0 CRON[15050]: (root) CMD (/usr/lib/sysstat/sadc 1 1 /var/log/sysstat/clearwater-sa`date +%d` > /dev/null 2>&1) Sep 29 06:41:03 homestead-0 monit: CMD /etc/init.d/homestead-prov restart Sep 29 06:41:03 homestead-0 monit: * Restarting homestead-prov homestead-prov Sep 29 06:41:03 homestead-0 homestead[15074]: 4005 - Description: Homestead started. @@Cause: The Homestead application is starting. @@Effect: Normal. @@Action: None. Sep 29 06:41:03 homestead-0 homestead[15074]: 1001 - Description: Diameter stack is starting. @@Cause: Diameter stack is beginning initialization. @@Effect: Normal. @@Action: None. Sep 29 06:41:03 homestead-0 homestead[15074]: 1002 - Description: Diameter stack initialization completed. @@Cause: Diameter stack has completed initialization. @@Effect: Normal. @@Action: None. Sep 29 06:41:03 homestead-0 homestead[15074]: 4007 - Description: Fatal - Failed to initialize Diameter stack in function fd_core_parseconf with error 2. @@Cause: The Diameter interface could not be initialized or encountered an error while running. @@Effect: The application will exit and restart until the problem is fixed. @@Action: (1). Check the configuration for the Diameter destination hosts. (2). Check the connectivity to the Diameter host using Wireshark. Sep 29 06:41:06 homestead-0 monit: Traceback (most recent call last): Sep 29 06:41:06 homestead-0 monit: File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main Sep 29 06:41:06 homestead-0 monit: "__main__", fname, loader, pkg_name) Sep 29 06:41:06 homestead-0 monit: File "/usr/lib/python2.7/runpy.py", line 72, in _run_code Sep 29 06:41:06 homestead-0 monit: exec code in run_globals Sep 29 06:41:06 homestead-0 monit: File "/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py", line 50, in <module> Sep 29 06:41:06 homestead-0 monit: from metaswitch.crest import api Sep 29 06:41:06 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/api/__init__.py", line 35, in <module> Sep 29 06:41:06 homestead-0 monit: from metaswitch.crest.api import base Sep 29 06:41:06 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/api/base.py", line 48, in <module> Sep 29 06:41:06 homestead-0 monit: from metaswitch.crest import settings Sep 29 06:41:06 homestead-0 monit: File "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/settings.py", line 139, in <module>
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
