[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

2017-09-11 Thread Graham Burgess
Additional information

List of nova packages installed on nova-cloud-controller:

$ dpkg -l | grep nova
ii  nova-api-os-compute  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - OpenStack Compute API frontend
ii  nova-cert2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - certificate management
ii  nova-common  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - common files
ii  nova-conductor   2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - conductor service
ii  nova-consoleauth 2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - Console Authenticator
ii  nova-novncproxy  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - NoVNC proxy
ii  nova-scheduler   2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - virtual machine scheduler
ii  python-nova  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute Python libraries
ii  python-novaclient2:3.3.1-2ubuntu1~cloud0
all  client library for OpenStack Compute API - Python 2.7

Keystone is configured for multi-domains, and there are 2 domains in
case that is pertinent, also their endpoints are not SSL:

$ openstack endpoint list --format csv -c "Service Name" -c "Service Type" -c 
"Interface" -c URL | grep keystone
"keystone","identity","internal","http://:5000/v3"
"keystone","identity","admin","http://:35357/v3"
"keystone","identity","public","http://:5000/v3"

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] [NEW] nova-novncproxy process gets wedged, requiring kill -HUP

2017-09-05 Thread Graham Burgess
Public bug reported:

Users reported they were unable to connect to instance consoles via
either Horizon or direct URL. Upon investigation we found errors
suggesting the address and port were in use:

2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] WebSocket 
server settings:
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - Listen 
on 0.0.0.0:6080
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - Flash 
security policy server
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - Web 
server (no directory listings). Web root: /usr/share/novnc
2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - No 
SSL/TLS support (no cert file)
2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address 
already in use
2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last):
2017-08-23 14:51:56.249 1355081 ERROR nova   File "/usr/bin/nova-novncproxy", 
line 10, in 
2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main())
2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main
2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port)
2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy
2017-08-23 14:51:56.249 1355081 ERROR nova 
RequestHandlerClass=websocketproxy.NovaProxyRequestHandler
2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in 
start_server
2017-08-23 14:51:56.249 1355081 ERROR nova tcp_keepintvl=self.tcp_keepintvl)
2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket
2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4])
2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/socket.py", line 224, in meth
2017-08-23 14:51:56.249 1355081 ERROR nova return 
getattr(self._sock,name)(*args)
2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already in 
use
2017-08-23 14:51:56.249 1355081 ERROR nova 

This lead us to the discovery of a stuck nova-novncproxy process after
stopping the service. Once we sent a kill -HUP to that process, we were
able to start the nova-novncproxy and restore service to the users.

This was not the first time we have had to restart nova-novncproxy
services after users reported that were unable to connect with VNC. This
time, as well as at least 2 other times, we have seen the following
errors in the nova-novncproxy.log during the time frame of the issue:

gaierror: [Errno -8] Servname not supported for ai_socktype

which seems to correspond to a log entries for connection strings with
an invalid port ('port': u'-1'). As well as a bunch of:

error: [Errno 104] Connection reset by peer

** Affects: nova (Ubuntu)
 Importance: Undecided
 Status: New


** Tags: canonical-bootstack

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs