Inline. Thanks, Kiall
On 21/07/15 09:24, Jaime Fernández wrote: > I confirm that it happened again. I started all the designate processes > and after an hour (approximately), the designate-api process died with > the same stack trace. > > Restarting the designate-api does not help because API requests are not > replied any more (timeouts): > 2015-07-21 10:12:53.463 4403 ERROR designate.api.middleware > [req-281e9665-c49b-43e0-a5d0-9a48e5f52aa1 noauth-user noauth-project - - > -] Timed out waiting for a reply to message ID > 19cda11c089f43e2a43a75a5851926c8 This suggests the API service is back up and running, but either your message broker (qpid), or the target service (designate-central) is down. > > When the designate-api process dies, I need to restart all the designate > processes. Then the api works correctly until the process dies again. This leads me to believe qpid is somehow responsible, but, I don't have any concrete reasons to believe that. Simply a gut feeling! If all services need to be restarted, then it's most likely a shared resource that's failing, mis-configured, or somehow incompatible with Designate. I've never seen anything like this before :/ > > I consider that it's normal the hex dump for qpid (it is debug level) > although I noticed it was different in rabbitmq. > > We have deployed designate in an Ubuntu host, as installation > instructions recommended, and I don't think there is any security issue > to stop the service. In fact, the trace is really strange because the > api was already bound on port 9001. Our Openstack platform is supported > by RedHat and that's why we need to integrate with qpid. > > I will try a couple of different scenarios: > a) Use a qpid local instance (instead of OST qpid instance) > b) Use a rabbitmq local instance Can you try with RabbitMQ? It's really the only one we test, but it's all hidden behind the oslo.messaging library, so I can't think of any good reason why one would work and the other doesn't. If this works, I'll test out with qpid and see if there's anything obvious. > > On Mon, Jul 20, 2015 at 5:20 PM, Kiall Mac Innes <[email protected] > <mailto:[email protected]>> wrote: > > Side Question: Is is normal for QPid to log all the > "\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00"'s etc? > > I'm guessing that, since you're using qpid, you're also on RedHat. Could > RH's SELinux policies be preventing the service from binding to > tcp/9001? > > If you start as root, do you see similar issues? > > Thanks, > Kiall > > On 20/07/15 15:48, Jaime Fernández wrote: > > Hi Tim, > > > > I only start one api process. In fact, when I say that the api process > > dies, I don't have any designate-api process and there is no process > > listening on the 9001 port. > > > > When I started all the designate processes, the API worked correctly > > because I had tested it. But after some inactivity period (a couple of > > hours, or a day), then the designate-api process died. It is not > > possible that the process has been restarted during this time. > > > > I've just started the process again and now it works. I will check if it > > dies again and report it. > > > > > > Thanks > > > > On Mon, Jul 20, 2015 at 4:24 PM, Tim Simmons <[email protected] > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > Jaime, > > > > > > Usually that's the error you see if you're trying to start up > > multiple API processes. They all try and bind to port 9001, so that > > error is saying the API can't bind. So something else (I suspect > > another designate-api process, or some other type of API) is already > > listening on that port. > > > > > > Hope that helps, > > > > Tim Simmons > > > > > > > ------------------------------------------------------------------------ > > *From:* Jaime Fernández <[email protected] > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>>> > > *Sent:* Monday, July 20, 2015 8:54 AM > > *To:* OpenStack Development Mailing List (not for usage questions) > > *Subject:* [openstack-dev] [designate] The designate API > service is > > stopped > > > > I've followed instructions to install Designate in Dev environment: > > > http://docs.openstack.org/developer/designate/install/ubuntu-dev.html > > > > I've made some slight modifications to use qpid (instead of > > rabbitmq) and to integrate with Infoblox. > > > > What I've seen is that designate-api process dies (the other > > processes are running correctly). I'm not sure if the problem > could > > be a network issue between designate-api and qpid. > > > > Here it is the output for last traces of designate-api process: > > > > 2015-07-20 14:43:37.728 727 DEBUG qpid.messaging.io.raw [-] > > READ[3f383f8]: > > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' > > readable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411 > > 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-] > > RCVD[3f383f8]: ConnectionHeartbeat() write > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651 > > 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-] > > SENT[3f383f8]: ConnectionHeartbeat() write_op > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683 > > 2015-07-20 14:43:37.730 727 DEBUG qpid.messaging.io.raw [-] > > SENT[3f383f8]: > > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00' > > writeable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475 > > Traceback (most recent call last): > > File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py", > > line 457, in fire_timers > > timer() > > File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/timer.py", > > line 58, in __call__ > > cb(*args, **kw) > > File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py", > > line 214, in main > > result = function(*args, **kwargs) > > File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py", > > line 623, in run_service > > service.start() > > File "/home/sysadmin/openstack/designate/designate/service.py", > > line 173, in start > > socket = self._wsgi_get_socket() > > File "/home/sysadmin/openstack/designate/designate/service.py", > > line 209, in _wsgi_get_socket > > 'port': self._service_config.api_port}) > > RuntimeError: Could not bind to 0.0.0.0:9001 > <http://0.0.0.0:9001> <http://0.0.0.0:9001> > > after trying for 30 seconds > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup [-] Could > > not bind to 0.0.0.0:9001 <http://0.0.0.0:9001> > <http://0.0.0.0:9001> after trying for 30 > > seconds > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > Traceback > > (most recent call last): > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py", > > line 154, in wait > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > x.wait() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py", > > line 51, in wait > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > return self.thread.wait() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py", > > line 175, in wait > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > return self._exit_event.wait() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/event.py", > > line 121, in wait > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > return hubs.get_hub().switch() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py", > > line 294, in switch > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > return self.greenlet.switch() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py", > > line 214, in main > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > result = function(*args, **kwargs) > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > > > "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py", > > line 623, in run_service > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > service.start() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > "/home/sysadmin/openstack/designate/designate/service.py", > line 173, > > in start > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > socket = self._wsgi_get_socket() > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File > > "/home/sysadmin/openstack/designate/designate/service.py", > line 209, > > in _wsgi_get_socket > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > 'port': self._service_config.api_port}) > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > RuntimeError: Could not bind to 0.0.0.0:9001 > <http://0.0.0.0:9001> <http://0.0.0.0:9001> > > after trying for 30 seconds > > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup > > 2015-07-20 14:43:55.222 727 INFO designate.service [-] > Stopping api > > service > > 2015-07-20 14:43:55.222 727 DEBUG qpid.messaging.io.ops [-] > > SENT[3f383f8]: > > SessionDetach(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31') > > write_op > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683 > > 2015-07-20 14:43:55.223 727 DEBUG qpid.messaging.io.raw [-] > > SENT[3f383f8]: > > > > "\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31" > > writeable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475 > > 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.raw [-] > > READ[3f383f8]: > > > > "\x0f\x00\x00:\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31\x00" > > readable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411 > > 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.ops [-] > > RCVD[3f383f8]: > > SessionDetached(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31', > > code=0) write > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651 > > 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.ops [-] > > SENT[3f383f8]: ConnectionClose(reply_code=200) write_op > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683 > > 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.raw [-] > > SENT[3f383f8]: > > > > '\x0f\x00\x00\x12\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0b\x01\x00\x00\xc8' > > writeable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475 > > 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.raw [-] > > READ[3f383f8]: > > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0c\x00\x00' > > readable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411 > > 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.ops [-] > > RCVD[3f383f8]: ConnectionCloseOk() write > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651 > > 2015-07-20 14:43:55.639 727 DEBUG qpid.messaging.io.ops [-] > > SENT[3ea43f8]: MessageCancel(destination='0', id=serial(71)) > > write_op > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683 > > 2015-07-20 14:43:55.640 727 DEBUG qpid.messaging.io.ops [-] > > SENT[3ea43f8]: > > QueueDelete(queue='reply_0b3c96ac8b0c4eabbce54791d02166ec', > > id=serial(72), sync=True) write_op > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683 > > 2015-07-20 14:43:55.641 727 DEBUG qpid.messaging.io.raw [-] > > SENT[3ea43f8]: > > > > '\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x04\x08\x01\x00\x01\x00\x010\x0f\x01\x009\x00\x01\x00\x00\x00\x00\x00\x00\x08\x02\x01\x01\x01\x00&reply_0b3c96ac8b0c4eabbce54791d02166ec' > > writeable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475 > > 2015-07-20 14:43:55.653 727 DEBUG qpid.messaging.io.raw [-] > > READ[3ea43f8]: > > > > '\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00H' > > readable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411 > > 2015-07-20 14:43:55.654 727 DEBUG qpid.messaging.io.ops [-] > > RCVD[3ea43f8]: SessionCompleted(commands=[0-72]) write > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651 > > 2015-07-20 14:43:55.655 727 DEBUG qpid.messaging.io.ops [-] > > SENT[3ea43f8]: > > SessionDetach(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0') > > write_op > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683 > > 2015-07-20 14:43:55.656 727 DEBUG qpid.messaging.io.raw [-] > > SENT[3ea43f8]: > > > > '\x0f\x00\x008\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0' > > writeable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475 > > 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.raw [-] > > READ[3ea43f8]: > > > > '\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0\x00' > > readable > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411 > > 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.ops [-] > > RCVD[3ea43f8]: > > SessionDetached(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0', > > code=0) write > > > > /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651 > > > > Have you experienced this issue? Do you know the source of the > problem? > > > > Thanks. > > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > > [email protected]?subject:unsubscribe > <http://[email protected]?subject:unsubscribe> > > > <http://[email protected]?subject:unsubscribe> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > [email protected]?subject:unsubscribe > <http://[email protected]?subject:unsubscribe> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > [email protected]?subject:unsubscribe > <http://[email protected]?subject:unsubscribe> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: [email protected]?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
