Hi, It looks like monit has lost track of the mysql process. It's constantly trying to restart mysql, but this is failing because there's already a mysql process running. Can you kill every mysql process on the node, and then allow monit to restart it?
Can you also please send me your app-servers.json file? Ellie From: Clearwater [mailto:[email protected]] On Behalf Of Dennis Diamsay Sent: 29 July 2015 11:00 To: [email protected] Subject: [Clearwater] Failed to configure a number to use Memento AS in Ellis UI and mysql_process status Execution Failed HI All, I have a manual install Clearwater with 5 virtual machine (except ralf) installed in VM Workstation. I have successfully created new numbers in Ellis and have a successful call between zoiper installed in laptop and zoiper installed in android. After that i enabled Memento in Sprout config file and edited /usr/share/clearwater/ellis/web-content/js/app-servers.json in Ellis. After selecting "Configure" button in Ellis UI, i received Failed to Update Server... error message. Then i tried to check on Ellis and 'mysql_process' have a status of Execution failed. Executed 'sudo service ellis stop' but no effect. I tried sudo reboot but still no effect. I tried recreating the Ellis node because i cant seem to make 'mysql_process' running. After recreating the Ellis node, 'sudo monit status' results are all Running. I again tried editing /usr/share/clearwater/ellis/web-content/js/app-servers.json in Ellis and same problem occured. Can you help me on how to configure a number to use Memento AS in Ellis UI? How can i troubleshoot 'mysql_process' with status Execution Failed? LOGS: Please see config file # Deployment definitions home_domain=example.com<http://example.com> sprout_hostname=192.168.1.101 hs_hostname=192.168.1.71:8888 hs_provisioning_hostname=192.168.1.71:8889 xdms_hostname=192.168.1.61:7888 ralf_hostname= #chronos_hostname=<private IP>:7253 # Local IP configuration local_ip=192.168.1.101 public_ip=192.168.1.101 public_hostname=192.168.1.101 # Email server configuration smtp_smarthost=127.0.0.1 smtp_username=username smtp_password=password [email protected]<mailto:[email protected]> # Keys signup_key=entropy turn_workaround_secret ellis_api_key=secret ellis_cookie_key=secret # I-CSCF/S-CSCF configuration #icscf=5052 #scscf=5054 upstream_hostname=192.168.1.101 # Application Servers gemini_enabled='Y' memento_enabled='Y' # ENUM #enum_server=<public IP> #enum_file=<file.json> # Additional domains #additional_home_domains=<domain 1>,<domain 2>,<domain 3>,... Please see some of /var/log/mysql/error.log in Ellis node. InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:42:36 InnoDB: Unable to open the first data file InnoDB: Error in opening ./ibdata1 150729 17:42:36 InnoDB: Operating system error number 11 in a file operation. InnoDB: Error number 11 means 'Resource temporarily unavailable'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html 150729 17:42:36 InnoDB: Could not open or create data files. 150729 17:42:36 InnoDB: If you tried to add new data files, and it failed here, 150729 17:42:36 InnoDB: you should now edit innodb_data_file_path in my.cnf back 150729 17:42:36 InnoDB: to what it was, and remove the new ibdata files InnoDB created 150729 17:42:36 InnoDB: in this failed attempt. InnoDB only wrote those files full of 150729 17:42:36 InnoDB: zeros, but did not yet use them in any way. But be careful: do not 150729 17:42:36 InnoDB: remove old data files which contain your precious data! 150729 17:42:36 [ERROR] Plugin 'InnoDB' init function returned error. 150729 17:42:36 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 150729 17:42:36 [ERROR] Unknown/unsupported storage engine: InnoDB 150729 17:42:36 [ERROR] Aborting 150729 17:42:36 [Note] /usr/sbin/mysqld: Shutdown complete InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:42:36 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:43:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 150729 17:43:36 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead. 150729 17:43:36 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1) starting as process 52002 ... 150729 17:43:36 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150729 17:43:36 [Note] Plugin 'FEDERATED' is disabled. 150729 17:43:36 InnoDB: The InnoDB memory heap is disabled 150729 17:43:36 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150729 17:43:36 InnoDB: Compressed tables use zlib 1.2.8 150729 17:43:36 InnoDB: Using Linux native AIO 150729 17:43:36 InnoDB: Initializing buffer pool, size = 128.0M 150729 17:43:36 InnoDB: Completed initialization of buffer pool InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:43:36 InnoDB: Retrying to lock the first data file InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:43:56 InnoDB: Unable to open the first data file InnoDB: Error in opening ./ibdata1 150729 17:43:56 InnoDB: Operating system error number 11 in a file operation. InnoDB: Error number 11 means 'Resource temporarily unavailable'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html 150729 17:43:56 InnoDB: Could not open or create data files. 150729 17:43:56 InnoDB: If you tried to add new data files, and it failed here, 150729 17:43:56 InnoDB: you should now edit innodb_data_file_path in my.cnf back 150729 17:43:56 InnoDB: to what it was, and remove the new ibdata files InnoDB created 150729 17:43:56 InnoDB: in this failed attempt. InnoDB only wrote those files full of 150729 17:43:56 InnoDB: zeros, but did not yet use them in any way. But be careful: do not 150729 17:43:56 InnoDB: remove old data files which contain your precious data! 150729 17:43:56 [ERROR] Plugin 'InnoDB' init function returned error. 150729 17:43:56 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 150729 17:43:56 [ERROR] Unknown/unsupported storage engine: InnoDB 150729 17:43:56 [ERROR] Aborting 150729 17:43:56 [Note] /usr/sbin/mysqld: Shutdown complete 150729 17:43:56 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:44:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 150729 17:44:16 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead. 150729 17:44:16 [Note] /usr/sbin/mysqld (mysqld 5.5.44-0ubuntu0.14.04.1) starting as process 52780 ... 150729 17:44:16 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 150729 17:44:16 [Note] Plugin 'FEDERATED' is disabled. 150729 17:44:16 InnoDB: The InnoDB memory heap is disabled 150729 17:44:16 InnoDB: Mutexes and rw_locks use GCC atomic builtins 150729 17:44:16 InnoDB: Compressed tables use zlib 1.2.8 150729 17:44:16 InnoDB: Using Linux native AIO 150729 17:44:16 InnoDB: Initializing buffer pool, size = 128.0M 150729 17:44:16 InnoDB: Completed initialization of buffer pool InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. 150729 17:44:16 InnoDB: Retrying to lock the first data file InnoDB: Unable to lock ./ibdata1, error: 11 InnoDB: Check that you do not already have another mysqld process InnoDB: using the same InnoDB data or log files. InnoDB: Unable to lock ./ibdata1, error: 11 Please see ps aux grep mysql [ellis]ellis@ellis:~$ ps aux | grep mysql mysql 1419 0.0 2.7 624148 54284 ? Ssl 16:58 0:01 /usr/sbin/mysqld root 55542 0.0 0.0 4448 1680 ? S 17:46 0:00 /bin/sh /usr/bin/mysqld_safe mysql 55889 0.0 1.9 381000 38816 ? Sl 17:46 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 root 56322 0.0 0.0 4448 1612 ? S 17:47 0:00 /bin/sh /usr/bin/mysqld_safe mysql 56669 0.0 1.8 381000 37800 ? Sl 17:47 0:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 ellis 57115 0.0 0.0 11724 1992 pts/1 S+ 17:48 0:00 grep --color=auto mysql [ellis]ellis@ellis:~$ Please see 'sudo monit status' [ellis]ellis@ellis:~$ sudo monit status The Monit daemon 5.8.1 uptime: 42m Process 'ntp_process' status Running monitoring status Monitored pid 2452 parent pid 1 uid 106 effective uid 106 gid 113 uptime 42m children 0 memory kilobytes 4328 memory kilobytes total 4328 memory percent 0.2% memory percent total 0.2% cpu percent 0.0% cpu percent total 0.0% data collected Wed, 29 Jul 2015 17:40:55 System 'node-ellis' status Running monitoring status Monitored load average [0.00] [0.02] [0.05] cpu 1.3%us 0.7%sy 0.0%wa memory usage 264176 kB [13.1%] swap usage 0 kB [0.0%] data collected Wed, 29 Jul 2015 17:40:55 Process 'nginx_process' status Running monitoring status Monitored pid 1465 parent pid 1 uid 0 effective uid 0 gid 0 uptime 42m children 4 memory kilobytes 2776 memory kilobytes total 17256 memory percent 0.1% memory percent total 0.8% cpu percent 0.0% cpu percent total 0.0% port response time 0.000s to 127.0.0.1:80/ping [HTTP via TCP] data collected Wed, 29 Jul 2015 17:40:55 Process 'mysql_process' status Execution failed monitoring status Monitored data collected Wed, 29 Jul 2015 17:40:45 Process 'ellis_process' status Running monitoring status Monitored pid 12856 parent pid 1 uid 1000 effective uid 1000 gid 1000 uptime 31m children 2 memory kilobytes 22120 memory kilobytes total 76452 memory percent 1.1% memory percent total 3.8% cpu percent 0.0% cpu percent total 0.0% data collected Wed, 29 Jul 2015 17:40:45 Program 'poll_ellis' status Status ok monitoring status Monitored last started Wed, 29 Jul 2015 17:40:45 last exit value 0 data collected Wed, 29 Jul 2015 17:40:45 Process 'clearwater_diags_monitor_process' status Running monitoring status Monitored pid 1445 parent pid 1 uid 0 effective uid 0 gid 0 uptime 42m children 1 memory kilobytes 3228 memory kilobytes total 3924 memory percent 0.1% memory percent total 0.1% cpu percent 0.0% cpu percent total 0.0% data collected Wed, 29 Jul 2015 17:40:45 [ellis]ellis@ellis:~$
_______________________________________________ Clearwater mailing list [email protected] http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
