Martin Kosek wrote:
Hello,

I was investigating global unit test failure on Fedora 18 for most of today, I
would like to share results I found so far.

Unit test and its related scripts on F18 now reports NSS BUSY exception, just
like this one:

# ./make-testcert
Traceback (most recent call last):
   File "./make-testcert", line 134, in <module>
     sys.exit(makecert(reqdir))
   File "./make-testcert", line 111, in makecert
     add=True)
   File "./make-testcert", line 68, in run
     result = self.execute(method, *args, **options)
   File "/root/freeipa-master2/ipalib/backend.py", line 146, in execute
     raise error #pylint: disable=E0702
ipalib.errors.NetworkError: cannot connect to
'http://vm-042.idm.lab.bos.redhat.com/ipa/session/xml': [Errno -8053]
(SEC_ERROR_BUSY) NSS could not shutdown. Objects are still in use.

Something In F18 must have changed, this worked before... But leaked
NSSConnection objects without proper close() now ends with the exception above.

In case of make-testcert script, the exception is raised because the script
does the following procedure:

1) connect, do one command
2) disconnect
3) connect, do second command

However, during disconnect, NSSConnection is leaked which makes NSS very
uncomfortable during second connection atempt (and nss_shutdown()). I managed
to fix this issue with attached patch. ./make-testcert or "./make-test
tests/test_xmlrpc/test_group_plugin.py" works fine now.

But global "./make-test" still fails, I think there is some remaining
NSSConnection leak, I suspect there is something wrong with how we use our
context (threading.local object). It looses a connection or some other thread
invoked in ldap2 module may be kicking in, here is my debug output:

CONTEXT[xmlclient] = <ipalib.request.Connection object at 0x9a1f5ec>

Test a simple LDAP bind using ldap2 ... SKIP: No directory manager password in
/root/.ipa/.dmpw
Test the `ipaserver.rpcserver.jsonserver.unmarshal` method. ... ok
tests.test_ipaserver.test_rpcserver.test_session.test_mount ... CONTEXT
150714476: GET languages

CONTEXT[xmlclient] = None

The connection is in the context, but then something happens and it is gone.
Then, unit tests try to connect again and NSS fails.

I would be really glad if somebody with a knowledge of NSS or how threads in
Python/IPA work could give me some advice...

Thanks!
Martin

I built upon your patch and have something that seems to work at least somewhat. I'm getting some unexpected test failures when running the entire suite but no NSS shutdown errors. I haven't had a chance to really investigate everything yet, sending this out as a work-in-progress in case you want to take a look.

rob

>From ad3f420395c8e4fc24d9ab3aa6f53641f188efc6 Mon Sep 17 00:00:00 2001
From: Rob Crittenden <rcrit...@redhat.com>
Date: Wed, 17 Oct 2012 16:58:54 -0400
Subject: [PATCH] candidate for fixing test execution against httpd

---
 ipalib/rpc.py       | 22 +++++++++++++++++++---
 ipapython/nsslib.py |  6 ++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

diff --git a/ipalib/rpc.py b/ipalib/rpc.py
index e97536d..6de45cf 100644
--- a/ipalib/rpc.py
+++ b/ipalib/rpc.py
@@ -257,6 +257,9 @@ class SSLTransport(LanguageAwareTransport):
         # If we an existing connection exists using the same NSS database
         # there is no need to re-initialize. Pass thsi into the NSS
         # connection creator.
+        if self._connection and host == self._connection[0]:
+            return self._connection[1]
+
         dbdir = '/etc/pki/nssdb'
         no_init = self.__nss_initialized(dbdir)
         (major, minor, micro, releaselevel, serial) = sys.version_info
@@ -265,8 +268,10 @@ class SSLTransport(LanguageAwareTransport):
         else:
             conn = NSSConnection(host, 443, dbdir=dbdir, no_init=no_init)
         self.dbdir=dbdir
+
         conn.connect()
-        return conn
+        self._connection = host, conn
+        return self._connection[1]
 
 
 class KerbTransport(SSLTransport):
@@ -331,6 +336,13 @@ class KerbTransport(SSLTransport):
 
         return (host, extra_headers, x509)
 
+
+    def single_request(self, host, handler, request_body, verbose=0):
+        try:
+            return SSLTransport.single_request(self, host, handler, request_body, verbose)
+        finally:
+            self.close()
+
     def parse_response(self, response):
         session_cookie = response.getheader('Set-Cookie')
         if session_cookie:
@@ -371,7 +383,8 @@ class xmlclient(Connectible):
         """
         if not hasattr(self.conn, '_ServerProxy__transport'):
             return None
-        if type(self.conn._ServerProxy__transport) in (KerbTransport, DelegatedKerbTransport):
+        if (isinstance(self.conn._ServerProxy__transport, KerbTransport) or
+            isinstance(self.conn._ServerProxy__transport, DelegatedKerbTransport)):
             scheme = "https"
         else:
             scheme = "http"
@@ -493,7 +506,10 @@ class xmlclient(Connectible):
         return serverproxy
 
     def destroy_connection(self):
-        pass
+        conn = getattr(context, self.id, None)
+        if conn is not None:
+            conn = conn.conn._ServerProxy__transport
+            conn.close()
 
     def forward(self, name, *args, **kw):
         """
diff --git a/ipapython/nsslib.py b/ipapython/nsslib.py
index 06bcba6..7afccd5 100644
--- a/ipapython/nsslib.py
+++ b/ipapython/nsslib.py
@@ -238,6 +238,12 @@ class NSSConnection(httplib.HTTPConnection, NSSAddressFamilyFallback):
     def connect(self):
         self.connect_socket(self.host, self.port)
 
+    def close(self):
+        """Close the connection to the HTTP server."""
+        if self.sock:
+            self.sock.close()   # close it manually... there may be other refs
+            self.sock = None
+
     def endheaders(self, message=None):
         """
         Explicitly close the connection if an error is returned after the
-- 
1.7.12.1

_______________________________________________
Freeipa-devel mailing list
Freeipa-devel@redhat.com
https://www.redhat.com/mailman/listinfo/freeipa-devel

Reply via email to