Hello,
I believe I have tracked down the issue. When setting up our Fibre Channel
CI, we created a script to listen to the gerrit event stream that was
using the Œnetapp-ci¹ account. This script had a code path where the
paramiko ssh connection would not be closed and this has been resolved.
Evidence that this was the culprit is that the script had been running
without interruption since April 29th and that is consistent with the
Gerrit connections from this account we see here:
http://paste.openstack.org/show/215173/
I apologize for the inconvenience and appreciate all the help in
#openstack-infra. I think it is safe for the account to be reenabled.
Thanks,
-Alex
On 5/5/15, 8:50 PM, James E. Blair cor...@inaugust.com wrote:
Hi,
We've been tracking a bug in Gerrit recently where all of the threads
tasked with servicing the stream-events command eventually get stuck.
This causes all of the CI systems, including OpenStack's, to stop
responding to events until the server is manually restarted.
We recently found that had happened with connections from the netapp-ci
account. I believe that Gerrit should be more resilient to these kinds
of errors, however, due to the severe impact to the project when this
happens, I have disabled the netapp-ci account until we find a solution
to the problem.
Note that the Gerrit upgrade scheduled for Saturday May 9 will bring a
new SSH server with it, and may have an impact on this issue.
If the netapp-ci operators have a moment to chat with us in
#openstack-infra on Freenode that would probably be the best way to work
on a plan to debug the problem further.
Thanks, and sorry for the inconvenience,
Jim
___
Third-party-announce mailing list
third-party-annou...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
