[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Edward Hope-Morley]

** Tags removed: verification-needed
** Tags added: sts sts-sru-done verification-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Corey Bryant]

This bug was fixed in the package websockify - 0.6.0+dfsg1-1~cloud2
---

 websockify (0.6.0+dfsg1-1~cloud2) trusty-kilo; urgency=medium
 .
   * Fix hanging nova-novncproxy and can't be restarted (LP: #1715254)
 - [PATCH] Make websockify respect SIGTERM
 - [PATCH] Remove additional signal calls in websockify that
   causes novnc to hang.


** Changed in: cloud-archive/kilo
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to websockify in Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Corey Bryant]

This bug was fixed in the package websockify - 0.6.0+dfsg1-1~cloud2
---

 websockify (0.6.0+dfsg1-1~cloud2) trusty-kilo; urgency=medium
 .
   * Fix hanging nova-novncproxy and can't be restarted (LP: #1715254)
 - [PATCH] Make websockify respect SIGTERM
 - [PATCH] Remove additional signal calls in websockify that
   causes novnc to hang.


** Changed in: cloud-archive/kilo
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

ii  websockify   0.6.0+dfsg1-1~cloud2
amd64WebSockets support for any application/server

reconnection test for several hours.

Thanks.

** Tags removed: verification-kilo-needed
** Tags added: verification-kilo-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Edward Hope-Morley]

** No longer affects: cloud-archive/icehouse

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** No longer affects: websockify (Ubuntu Trusty)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Changed in: cloud-archive/kilo
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

** Changed in: cloud-archive/mitaka
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

** Changed in: websockify (Ubuntu Trusty)
 Assignee: Seyeong Kim (xtrusia) => (unassigned)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[James Page]

This bug was fixed in the package websockify - 0.6.1+dfsg1-1ubuntu1~cloud0
---

 websockify (0.6.1+dfsg1-1ubuntu1~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 websockify (0.6.1+dfsg1-1ubuntu1) xenial; urgency=medium
 .
   * Fix hanging nova-novncproxy and can't be restarted (LP: #1715254)
 - [PATCH] Make websockify respect SIGTERM
 - [PATCH] Remove additional signal calls in websockify that
   causes novnc to hang.


** Changed in: cloud-archive/mitaka
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to websockify in Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[James Page]

This bug was fixed in the package websockify - 0.6.1+dfsg1-1ubuntu1~cloud0
---

 websockify (0.6.1+dfsg1-1ubuntu1~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 websockify (0.6.1+dfsg1-1ubuntu1) xenial; urgency=medium
 .
   * Fix hanging nova-novncproxy and can't be restarted (LP: #1715254)
 - [PATCH] Make websockify respect SIGTERM
 - [PATCH] Remove additional signal calls in websockify that
   causes novnc to hang.


** Changed in: cloud-archive/mitaka
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

upgraded to 0.6.1+dfsg1-1ubuntu1~cloud0

tested same steps as above.

it works fine.

Thanks.


** Tags removed: verification-mitaka-needed
** Tags added: verification-mitaka-done

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Corey Bryant]

Hello Seyeong,

This is all set now. We had an issue with the cloud archive sync. Can
you try again?

Thanks,
Corey

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

hello corey

I checked trusty-proposed/mitaka/main/binary-amd64/Packages

but websockify version is

0.6.1+dfsg1-1~cloud1

but it is current version i think.

you need to check this?

Thanks

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

it seems not in -proposed yet,

I'll test this when I can upgrade websockify

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Launchpad Bug Tracker]

This bug was fixed in the package websockify - 0.6.1+dfsg1-1ubuntu1

---
websockify (0.6.1+dfsg1-1ubuntu1) xenial; urgency=medium

  * Fix hanging nova-novncproxy and can't be restarted (LP: #1715254)
- [PATCH] Make websockify respect SIGTERM
- [PATCH] Remove additional signal calls in websockify that
  causes novnc to hang.

 -- Seyeong Kim   Mon, 23 Oct 2017 18:31:40
+0900

** Changed in: websockify (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

Hello,

I tested this proposed pkg and confirmed it is working fine.

For testing, I just did steps on [Test case] section.

1. juju deploy xenial.bundle
2. create network & subnet
3. juju config nova-cloud-controller console-access-protocol=novnc
4. create instance
5. refreshing every 5 seconds on 2 browsers with console url for several hors

Thanks

ii  websockify   0.6.1+dfsg1-1ubuntu1
amd64WebSockets support for any application/server

** Tags removed: verification-needed-xenial
** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Brian Murray]

Hello Graham, or anyone else affected,

Accepted websockify into xenial-proposed. The package will build now and
be available at
https://launchpad.net/ubuntu/+source/websockify/0.6.1+dfsg1-1ubuntu1 in
a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-xenial to verification-done-xenial. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-xenial. In either case, details of your
testing will help us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: websockify (Ubuntu Xenial)
   Status: Triaged => Fix Committed

** Tags added: verification-needed verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

Hello Corey,
I've tested Trusty & UCA Icehouse.
However, I couldn't reproduce this issue. 
msgs in logs are different to kilo, mitaka, xenial

There is no 'Reaing zombies, active child count is'.
There are a lot of them on kilo, mitaka, xenial

I saw jame's latest commit which is patch for multiprocessing
but it seems not working on trusty, uca icehouse ( not sure 100% )

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Description changed:

  [Impact]
  
  affected
- - UCA Mitaka
+ - UCA Mitaka, Kilo
  - Xenial
  
- need to test
+ 
+ not affected
  - UCA Icehouse
  - Trusty
+ ( log symptom is different, there is no reaing(which is errata) zombie... etc)
  
- TBD
  
  [Test case]
  
  1. Deploy openstack
  2. Creating instances
  3. open console in browser with auto refresh extension ( set 5 seconds )
  4. after several hours connection rejected
- 
- TBD
  
  [Regression Potential]
  
  Components that using websockify, escpecially nova-novncproxy, will be
  affected by this patch. However, After upgrading this and refreshing
  test above mentioned for 2 days without restarting any services, no hang
  happens. I tested this test in my local simple environment, so need to
  be considered possibility in different circumstances.
  
  [Others]
  
  related commits
  
  - https://github.com/novnc/websockify/pull/226
  - https://github.com/novnc/websockify/pull/219
  
  [Original Description]
  
- 
- Users reported they were unable to connect to instance consoles via either 
Horizon or direct URL. Upon investigation we found errors suggesting the 
address and port were in use:
+ Users reported they were unable to connect to instance consoles via
+ either Horizon or direct URL. Upon investigation we found errors
+ suggesting the address and port were in use:
  
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] 
WebSocket server settings:
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - 
Listen on 0.0.0.0:6080
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - 
Flash security policy server
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - Web 
server (no directory listings). Web root: /usr/share/novnc
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - No 
SSL/TLS support (no cert file)
  2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address 
already in use
  2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last):
  2017-08-23 14:51:56.249 1355081 ERROR nova   File "/usr/bin/nova-novncproxy", 
line 10, in 
  2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main())
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main
  2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port)
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy
  2017-08-23 14:51:56.249 1355081 ERROR nova 
RequestHandlerClass=websocketproxy.NovaProxyRequestHandler
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in 
start_server
  2017-08-23 14:51:56.249 1355081 ERROR nova 
tcp_keepintvl=self.tcp_keepintvl)
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket
  2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4])
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/socket.py", line 224, in meth
  2017-08-23 14:51:56.249 1355081 ERROR nova return 
getattr(self._sock,name)(*args)
  2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already 
in use
  2017-08-23 14:51:56.249 1355081 ERROR nova
  
  This lead us to the discovery of a stuck nova-novncproxy process after
  stopping the service. Once we sent a kill -HUP to that process, we were
  able to start the nova-novncproxy and restore service to the users.
  
  This was not the first time we have had to restart nova-novncproxy
  services after users reported that were unable to connect with VNC. This
  time, as well as at least 2 other times, we have seen the following
  errors in the nova-novncproxy.log during the time frame of the issue:
  
  gaierror: [Errno -8] Servname not supported for ai_socktype
  
  which seems to correspond to a log entries for connection strings with
  an invalid port ('port': u'-1'). As well as a bunch of:
  
  error: [Errno 104] Connection reset by peer

** Description changed:

  [Impact]
  
  affected
  - UCA Mitaka, Kilo
  - Xenial
  
- 
  not affected
  - UCA Icehouse
  - Trusty
  ( log symptom is different, there is no reaing(which is errata) zombie... etc)
  
+ When number of connections are many or frequently reconnecting to
+ console, nova-novncproxy daemon is stuck because websockify is hang.
  
  [Test case]
  
  1. Deploy openstack
  2. Creating instances
  3. open console in browser with auto refresh extension ( set 5 seconds )
  4. after several hours connection rejected
  
  [Regression Potential]
  
  Components that using websockify, escpecially nova-novncproxy, will be
  affected by this patch. However, After upgrading this and refreshing
  test above mentioned for 2 days without restarting any services, no hang
 

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

Hello Corey,

I've uploaded patch for kilo.

I'm going to upload patches for icehouse and trusty

after testing them.

I'm testing them but log is little different.

will keep posting

Thanks

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Patch added: "lp1715254-kilo.debdiff"
   
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+attachment/4998576/+files/lp1715254-kilo.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Corey Bryant]

I've uploaded Seyeong's xenial patch to the xenial review queue and it is 
awaiting SRU team review.
https://launchpad.net/ubuntu/xenial/+queue?queue_state=1_text=

If you'd like to provide patches for trusty-kilo and trusty-icehouse I'd
be happy to sponsor those as well.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Corey Bryant]

Thanks for the patches Seyeong. Assuming those fix the problem this only
affects websockify < 0.8.0, which are releases prior to Yakkety/Newton.

** Also affects: cloud-archive/kilo
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/icehouse
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/mitaka
   Importance: Undecided
   Status: New

** Changed in: cloud-archive/kilo
   Status: New => Triaged

** Changed in: cloud-archive/icehouse
   Status: New => Triaged

** Changed in: cloud-archive/icehouse
   Importance: Undecided => Medium

** Changed in: cloud-archive/kilo
   Importance: Undecided => Medium

** Changed in: cloud-archive/mitaka
   Importance: Undecided => Medium

** Changed in: cloud-archive
   Status: New => Invalid

** Changed in: cloud-archive/mitaka
   Status: New => Triaged

** Also affects: websockify (Ubuntu Trusty)
   Importance: Undecided
   Status: New

** Also affects: websockify (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: websockify (Ubuntu Trusty)
   Status: New => Triaged

** Changed in: websockify (Ubuntu Xenial)
   Status: New => Triaged

** Changed in: websockify (Ubuntu Trusty)
   Importance: Undecided => Medium

** Changed in: websockify (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: websockify (Ubuntu Trusty)
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

** Changed in: websockify (Ubuntu Xenial)
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

** Changed in: websockify (Ubuntu)
   Status: Confirmed => Invalid

** Changed in: websockify (Ubuntu)
 Assignee: Seyeong Kim (xtrusia) => (unassigned)

** Changed in: websockify (Ubuntu)
   Importance: Medium => Undecided

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to websockify in Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Corey Bryant]

Thanks for the patches Seyeong. Assuming those fix the problem this only
affects websockify < 0.8.0, which are releases prior to Yakkety/Newton.

** Also affects: cloud-archive/kilo
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/icehouse
   Importance: Undecided
   Status: New

** Also affects: cloud-archive/mitaka
   Importance: Undecided
   Status: New

** Changed in: cloud-archive/kilo
   Status: New => Triaged

** Changed in: cloud-archive/icehouse
   Status: New => Triaged

** Changed in: cloud-archive/icehouse
   Importance: Undecided => Medium

** Changed in: cloud-archive/kilo
   Importance: Undecided => Medium

** Changed in: cloud-archive/mitaka
   Importance: Undecided => Medium

** Changed in: cloud-archive
   Status: New => Invalid

** Changed in: cloud-archive/mitaka
   Status: New => Triaged

** Also affects: websockify (Ubuntu Trusty)
   Importance: Undecided
   Status: New

** Also affects: websockify (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: websockify (Ubuntu Trusty)
   Status: New => Triaged

** Changed in: websockify (Ubuntu Xenial)
   Status: New => Triaged

** Changed in: websockify (Ubuntu Trusty)
   Importance: Undecided => Medium

** Changed in: websockify (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: websockify (Ubuntu Trusty)
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

** Changed in: websockify (Ubuntu Xenial)
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

** Changed in: websockify (Ubuntu)
   Status: Confirmed => Invalid

** Changed in: websockify (Ubuntu)
 Assignee: Seyeong Kim (xtrusia) => (unassigned)

** Changed in: websockify (Ubuntu)
   Importance: Medium => Undecided

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Ubuntu Foundations Team Bug Bot]

The attachment "lp1715254-mitaka.debdiff" seems to be a debdiff.  The
ubuntu-sponsors team has been subscribed to the bug report so that they
can review and hopefully sponsor the debdiff.  If the attachment isn't a
patch, please remove the "patch" flag from the attachment, remove the
"patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe
the team.

[This is an automated message performed by a Launchpad user owned by
~brian-murray, for any issue please contact him.]

** Tags added: patch

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Description changed:

- Users reported they were unable to connect to instance consoles via
- either Horizon or direct URL. Upon investigation we found errors
- suggesting the address and port were in use:
+ [Impact]
+ 
+ affected
+ - UCA Mitaka
+ - Xenial
+ 
+ need to test
+ - UCA Icehouse
+ - Trusty
+ 
+ TBD
+ 
+ [Test case]
+ 
+ 1. Deploy openstack
+ 2. Creating instances
+ 3. open console in browser with auto refresh extension ( set 5 seconds )
+ 4. after several hours connection rejected
+ 
+ TBD
+ 
+ [Regression Potential]
+ 
+ Components that using websockify, escpecially nova-novncproxy, will be
+ affected by this patch. However, After upgrading this and refreshing
+ test above mentioned for 2 days without restarting any services, no hang
+ happens. I tested this test in my local simple environment, so need to
+ be considered possibility in different circumstances.
+ 
+ [Others]
+ 
+ related commits
+ 
+ - https://github.com/novnc/websockify/pull/226
+ - https://github.com/novnc/websockify/pull/219
+ 
+ [Original Description]
+ 
+ 
+ Users reported they were unable to connect to instance consoles via either 
Horizon or direct URL. Upon investigation we found errors suggesting the 
address and port were in use:
  
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-] 
WebSocket server settings:
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - 
Listen on 0.0.0.0:6080
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - 
Flash security policy server
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - Web 
server (no directory listings). Web root: /usr/share/novnc
  2017-08-23 14:51:56.248 1355081 INFO nova.console.websocketproxy [-]   - No 
SSL/TLS support (no cert file)
  2017-08-23 14:51:56.249 1355081 CRITICAL nova [-] error: [Errno 98] Address 
already in use
  2017-08-23 14:51:56.249 1355081 ERROR nova Traceback (most recent call last):
  2017-08-23 14:51:56.249 1355081 ERROR nova   File "/usr/bin/nova-novncproxy", 
line 10, in 
  2017-08-23 14:51:56.249 1355081 ERROR nova sys.exit(main())
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/nova/cmd/novncproxy.py", line 41, in main
  2017-08-23 14:51:56.249 1355081 ERROR nova port=CONF.vnc.novncproxy_port)
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/nova/cmd/baseproxy.py", line 73, in proxy
  2017-08-23 14:51:56.249 1355081 ERROR nova 
RequestHandlerClass=websocketproxy.NovaProxyRequestHandler
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 909, in 
start_server
  2017-08-23 14:51:56.249 1355081 ERROR nova 
tcp_keepintvl=self.tcp_keepintvl)
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/dist-packages/websockify/websocket.py", line 698, in socket
  2017-08-23 14:51:56.249 1355081 ERROR nova sock.bind(addrs[0][4])
  2017-08-23 14:51:56.249 1355081 ERROR nova   File 
"/usr/lib/python2.7/socket.py", line 224, in meth
  2017-08-23 14:51:56.249 1355081 ERROR nova return 
getattr(self._sock,name)(*args)
  2017-08-23 14:51:56.249 1355081 ERROR nova error: [Errno 98] Address already 
in use
- 2017-08-23 14:51:56.249 1355081 ERROR nova 
+ 2017-08-23 14:51:56.249 1355081 ERROR nova
  
  This lead us to the discovery of a stuck nova-novncproxy process after
  stopping the service. Once we sent a kill -HUP to that process, we were
  able to start the nova-novncproxy and restore service to the users.
  
  This was not the first time we have had to restart nova-novncproxy
  services after users reported that were unable to connect with VNC. This
  time, as well as at least 2 other times, we have seen the following
  errors in the nova-novncproxy.log during the time frame of the issue:
  
  gaierror: [Errno -8] Servname not supported for ai_socktype
  
  which seems to correspond to a log entries for connection strings with
  an invalid port ('port': u'-1'). As well as a bunch of:
  
  error: [Errno 104] Connection reset by peer

** Patch added: "lp1715254-mitaka.debdiff"
   
https://bugs.launchpad.net/cloud-archive/+bug/1715254/+attachment/4990058/+files/lp1715254-mitaka.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Patch added: "lp1715254-xenial.debdiff"
   
https://bugs.launchpad.net/cloud-archive/+bug/1715254/+attachment/4990059/+files/lp1715254-xenial.debdiff

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Also affects: cloud-archive
   Importance: Undecided
   Status: New

** No longer affects: nova (Ubuntu)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Seyeong Kim]

** Changed in: websockify (Ubuntu)
 Assignee: (unassigned) => Seyeong Kim (xtrusia)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[James Page]

** Changed in: nova (Ubuntu)
   Importance: Undecided => Medium

** Changed in: websockify (Ubuntu)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: websockify (Ubuntu)
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Launchpad Bug Tracker]

Status changed to 'Confirmed' because the bug affects multiple users.

** Changed in: nova (Ubuntu)
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[James Page]

** Also affects: nova (Ubuntu)
   Importance: Undecided
   Status: New

** Also affects: websockify (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: charm-nova-cloud-controller
   Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to websockify in Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[James Page]

** Also affects: nova (Ubuntu)
   Importance: Undecided
   Status: New

** Also affects: websockify (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: charm-nova-cloud-controller
   Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Jill Rouleau]

** Package changed: nova (Ubuntu) => charm-nova-cloud-controller

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-nova-cloud-controller/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1715254] Re: nova-novncproxy process gets wedged, requiring kill -HUP

[Graham Burgess]

Additional information

List of nova packages installed on nova-cloud-controller:

$ dpkg -l | grep nova
ii  nova-api-os-compute  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - OpenStack Compute API frontend
ii  nova-cert2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - certificate management
ii  nova-common  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - common files
ii  nova-conductor   2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - conductor service
ii  nova-consoleauth 2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - Console Authenticator
ii  nova-novncproxy  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - NoVNC proxy
ii  nova-scheduler   2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute - virtual machine scheduler
ii  python-nova  2:13.1.4-0ubuntu2~cloud0   
all  OpenStack Compute Python libraries
ii  python-novaclient2:3.3.1-2ubuntu1~cloud0
all  client library for OpenStack Compute API - Python 2.7

Keystone is configured for multi-domains, and there are 2 domains in
case that is pertinent, also their endpoints are not SSL:

$ openstack endpoint list --format csv -c "Service Name" -c "Service Type" -c 
"Interface" -c URL | grep keystone
"keystone","identity","internal","http://:5000/v3"
"keystone","identity","admin","http://:35357/v3"
"keystone","identity","public","http://:5000/v3"

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1715254

Title:
  nova-novncproxy process gets wedged, requiring kill -HUP

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1715254/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs