Martin Kosek wrote:
On 10/18/2012 12:04 AM, Rob Crittenden wrote:
Martin Kosek wrote:
Hello,

I was investigating global unit test failure on Fedora 18 for most of today, I
would like to share results I found so far.

Unit test and its related scripts on F18 now reports NSS BUSY exception, just
like this one:

# ./make-testcert
Traceback (most recent call last):
    File "./make-testcert", line 134, in <module>
      sys.exit(makecert(reqdir))
    File "./make-testcert", line 111, in makecert
      add=True)
    File "./make-testcert", line 68, in run
      result = self.execute(method, *args, **options)
    File "/root/freeipa-master2/ipalib/backend.py", line 146, in execute
      raise error #pylint: disable=E0702
ipalib.errors.NetworkError: cannot connect to
'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053]
(SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.

Something In F18 must have changed, this worked before... But leaked
NSSConnection objects without proper close() now ends with the exception above.

In case of make-testcert script, the exception is raised because the script
does the following procedure:

1) connect, do one command
2) disconnect
3) connect, do second command

However, during disconnect, NSSConnection is leaked which makes NSS very
uncomfortable during second connection atempt (and nss_shutdown()). I managed
to fix this issue with attached patch. ./make-testcert or "./make-test
tests/test_xmlrpc/test_group_plugin.py" works fine now.

But global "./make-test" still fails, I think there is some remaining
NSSConnection leak, I suspect there is something wrong with how we use our
context (threading.local object). It looses a connection or some other thread
invoked in ldap2 module may be kicking in, here is my debug output:

CONTEXT[xmlclient] = <ipalib.request.Connection object at 0x9a1f5ec>

Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in
/root/.ipa/.dmpw
Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok
tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT
150714476: GET languages

CONTEXT[xmlclient] = None

The connection is in the context, but then something happens and it is gone.
Then, unit tests try to connect again and NSS fails.

I would be really glad if somebody with a knowledge of NSS or how threads in
Python/IPA work could give me some advice...

Thanks!
Martin

I built upon your patch and have something that seems to work at least
somewhat. I'm getting some unexpected test failures when running the entire
suite but no NSS shutdown errors. I haven't had a chance to really investigate
everything yet, sending this out as a work-in-progress in case you want to take
a look.

rob


Yeah, this is great! I tested a fresh build+install on Fedora 18 with your
patch and all tests succeeded. So as for F18, I am inclined to ACK the patch as 
is.

I am just not sure that this will work on platforms with Python version < 2.7,
xmlrpclib is different there.

Martin


Here is my first crack at fixing that too. It requires a bunch of run time juggling though.

rob
>From c53e283986f2b00db53e28009829ba09d62930aa Mon Sep 17 00:00:00 2001
From: Rob Crittenden <rcrit...@redhat.com>
Date: Wed, 17 Oct 2012 16:58:54 -0400
Subject: [PATCH] Close connection after each request, avoid NSS shutdown
 problem.

The unit tests were failing when executed against an Apache server
in F-18 due to dangling references causing NSS shutdown to fail.
---
 ipalib/rpc.py       | 30 +++++++++++++++++++++++++-----
 ipapython/nsslib.py |  6 ++++++
 2 files changed, 31 insertions(+), 5 deletions(-)

diff --git a/ipalib/rpc.py b/ipalib/rpc.py
index e97536d9de5c455d3ff58c081fca37f16d087370..8389396e0e23623b5edb60d634041949f95711ce 100644
--- a/ipalib/rpc.py
+++ b/ipalib/rpc.py
@@ -257,16 +257,24 @@ class SSLTransport(LanguageAwareTransport):
         # If we an existing connection exists using the same NSS database
         # there is no need to re-initialize. Pass thsi into the NSS
         # connection creator.
+        if sys.version_info > (2, 6):
+            if self._connection and host == self._connection[0]:
+                return self._connection[1]
+
         dbdir = '/etc/pki/nssdb'
         no_init = self.__nss_initialized(dbdir)
-        (major, minor, micro, releaselevel, serial) = sys.version_info
-        if major == 2 and minor < 7:
+        if sys.version_info < (2, 7):
             conn = NSSHTTPS(host, 443, dbdir=dbdir, no_init=no_init)
         else:
             conn = NSSConnection(host, 443, dbdir=dbdir, no_init=no_init)
         self.dbdir=dbdir
+
         conn.connect()
-        return conn
+        if sys.version_info < (2, 7):
+            return conn
+        else:
+            self._connection = host, conn
+            return self._connection[1]
 
 
 class KerbTransport(SSLTransport):
@@ -331,6 +339,13 @@ class KerbTransport(SSLTransport):
 
         return (host, extra_headers, x509)
 
+
+    def single_request(self, host, handler, request_body, verbose=0):
+        try:
+            return SSLTransport.single_request(self, host, handler, request_body, verbose)
+        finally:
+            self.close()
+
     def parse_response(self, response):
         session_cookie = response.getheader('Set-Cookie')
         if session_cookie:
@@ -371,7 +386,8 @@ class xmlclient(Connectible):
         """
         if not hasattr(self.conn, '_ServerProxy__transport'):
             return None
-        if type(self.conn._ServerProxy__transport) in (KerbTransport, DelegatedKerbTransport):
+        if (isinstance(self.conn._ServerProxy__transport, KerbTransport) or
+            isinstance(self.conn._ServerProxy__transport, DelegatedKerbTransport)):
             scheme = "https"
         else:
             scheme = "http"
@@ -493,7 +509,11 @@ class xmlclient(Connectible):
         return serverproxy
 
     def destroy_connection(self):
-        pass
+        if sys.version_info > (2, 6):
+            conn = getattr(context, self.id, None)
+            if conn is not None:
+                conn = conn.conn._ServerProxy__transport
+                conn.close()
 
     def forward(self, name, *args, **kw):
         """
diff --git a/ipapython/nsslib.py b/ipapython/nsslib.py
index 06bcba64895b0ba7a6b814ed6748eff8bf5ff9b3..7afccd5685baccdb8e9eff737cb7dd4b11d46630 100644
--- a/ipapython/nsslib.py
+++ b/ipapython/nsslib.py
@@ -238,6 +238,12 @@ class NSSConnection(httplib.HTTPConnection, NSSAddressFamilyFallback):
     def connect(self):
         self.connect_socket(self.host, self.port)
 
+    def close(self):
+        """Close the connection to the HTTP server."""
+        if self.sock:
+            self.sock.close()   # close it manually... there may be other refs
+            self.sock = None
+
     def endheaders(self, message=None):
         """
         Explicitly close the connection if an error is returned after the
-- 
1.7.12.1

_______________________________________________
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel

Reply via email to